Nov 02 01:11:59.598529 ip-10-0-136-130 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 02 01:11:59.598545 ip-10-0-136-130 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 02 01:11:59.598554 ip-10-0-136-130 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 02 01:11:59.598874 ip-10-0-136-130 systemd[1]: Failed to start Kubernetes Kubelet. Nov 02 01:12:09.687694 ip-10-0-136-130 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 02 01:12:09.687711 ip-10-0-136-130 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 6b66e05d81a84da7b7be4c59ed138df8 -- Nov 02 01:14:14.662669 ip-10-0-136-130 systemd[1]: Starting Kubernetes Kubelet... Nov 02 01:14:15.119401 ip-10-0-136-130 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 01:14:15.119401 ip-10-0-136-130 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 01:14:15.119401 ip-10-0-136-130 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 01:14:15.119401 ip-10-0-136-130 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 01:14:15.125039 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.124940 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 01:14:15.125039 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125032 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 01:14:15.125039 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125038 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 01:14:15.125039 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125042 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 01:14:15.125039 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125046 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 01:14:15.125039 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125050 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 01:14:15.125406 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125054 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 01:14:15.125406 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125058 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 01:14:15.125406 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125062 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 01:14:15.125406 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125065 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 01:14:15.125406 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125069 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 01:14:15.125406 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125073 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 01:14:15.125406 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125076 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 01:14:15.125406 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125090 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 01:14:15.125406 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125095 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 01:14:15.125406 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125101 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 01:14:15.125406 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125105 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 01:14:15.125406 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125109 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 01:14:15.125406 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125112 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 01:14:15.125406 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125116 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 02 01:14:15.125406 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125120 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 01:14:15.125406 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125124 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 01:14:15.125406 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125128 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 01:14:15.125406 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125132 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 01:14:15.125406 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125136 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 01:14:15.125406 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125140 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 01:14:15.126206 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125143 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 01:14:15.126206 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125147 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 01:14:15.126206 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125151 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 01:14:15.126206 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125155 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 01:14:15.126206 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125159 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 01:14:15.126206 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125163 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 01:14:15.126206 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125166 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 01:14:15.126206 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125170 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 01:14:15.126206 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125174 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 01:14:15.126206 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125178 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 01:14:15.126206 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125183 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 01:14:15.126206 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125188 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 01:14:15.126206 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125192 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 01:14:15.126206 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125197 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 01:14:15.126206 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125205 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 01:14:15.126206 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125212 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 01:14:15.126206 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125218 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 01:14:15.126206 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125222 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 01:14:15.126206 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125227 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 01:14:15.127027 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125233 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 01:14:15.127027 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125238 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 01:14:15.127027 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125243 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 01:14:15.127027 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125247 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 01:14:15.127027 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125251 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 01:14:15.127027 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125256 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 01:14:15.127027 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125260 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 01:14:15.127027 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125264 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 01:14:15.127027 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125268 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 01:14:15.127027 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125273 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 01:14:15.127027 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125305 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 01:14:15.127027 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125313 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 01:14:15.127027 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125318 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 01:14:15.127027 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125322 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 01:14:15.127027 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125326 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 01:14:15.127027 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125332 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 01:14:15.127027 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125336 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 01:14:15.127027 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125340 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 01:14:15.127027 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125344 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 01:14:15.127836 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125349 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 01:14:15.127836 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125353 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 01:14:15.127836 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125357 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 01:14:15.127836 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125361 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 01:14:15.127836 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125365 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 01:14:15.127836 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125387 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 01:14:15.127836 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125393 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 01:14:15.127836 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125398 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 01:14:15.127836 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125402 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 01:14:15.127836 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125406 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 01:14:15.127836 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125410 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 01:14:15.127836 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125414 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 01:14:15.127836 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125418 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 01:14:15.127836 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125422 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 01:14:15.127836 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125426 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 01:14:15.127836 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125436 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 01:14:15.127836 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125441 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 01:14:15.127836 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125445 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 01:14:15.127836 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125450 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 01:14:15.127836 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125454 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 01:14:15.128620 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125458 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 01:14:15.128620 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125462 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 01:14:15.128620 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125466 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 01:14:15.128620 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125471 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 01:14:15.128620 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125475 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 01:14:15.128620 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125480 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 01:14:15.128620 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125484 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 01:14:15.128620 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125489 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 01:14:15.128620 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125493 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 01:14:15.128620 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125497 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 01:14:15.128620 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125502 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 01:14:15.128620 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125506 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 01:14:15.128620 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125510 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 01:14:15.128620 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125514 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 01:14:15.128620 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125518 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 01:14:15.128620 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125522 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 01:14:15.128620 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125527 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 01:14:15.128620 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125531 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 01:14:15.128620 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125536 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 01:14:15.128620 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125542 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 01:14:15.129359 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125546 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 01:14:15.129359 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.125554 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 01:14:15.129359 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.126893 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 01:14:15.129359 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.126906 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 01:14:15.129359 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.126911 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 01:14:15.129359 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.126916 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 01:14:15.129359 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.126920 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 01:14:15.129359 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.126925 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 01:14:15.129359 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.126930 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 01:14:15.129359 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.126934 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 01:14:15.129359 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.126938 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 01:14:15.129359 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.126942 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 01:14:15.129359 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.126947 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 01:14:15.129359 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.126951 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 01:14:15.129359 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.126955 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 01:14:15.129359 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.126959 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 01:14:15.129359 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.126964 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 01:14:15.129359 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.126968 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 01:14:15.129359 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.126972 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 01:14:15.129837 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.126976 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 01:14:15.129837 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.126980 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 01:14:15.129837 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.126985 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 01:14:15.129837 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.126989 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 01:14:15.129837 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.126995 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 01:14:15.129837 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.126999 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 01:14:15.129837 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127003 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 01:14:15.129837 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127008 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 01:14:15.129837 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127012 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 01:14:15.129837 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127019 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 01:14:15.129837 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127025 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 01:14:15.129837 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127029 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 01:14:15.129837 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127034 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 01:14:15.129837 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127039 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 01:14:15.129837 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127043 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 01:14:15.129837 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127047 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 01:14:15.129837 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127051 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 01:14:15.129837 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127056 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 01:14:15.129837 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127060 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 01:14:15.130283 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127064 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 01:14:15.130283 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127068 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 01:14:15.130283 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127073 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 01:14:15.130283 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127078 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 01:14:15.130283 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127083 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 01:14:15.130283 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127087 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 01:14:15.130283 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127091 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 01:14:15.130283 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127095 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 01:14:15.130283 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127099 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 01:14:15.130283 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127104 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 01:14:15.130283 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127108 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 01:14:15.130283 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127112 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 01:14:15.130283 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127117 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 01:14:15.130283 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127121 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 01:14:15.130283 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127125 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 01:14:15.130283 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127129 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 01:14:15.130283 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127133 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 01:14:15.130283 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127138 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 01:14:15.130283 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127142 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 02 01:14:15.130770 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127146 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 01:14:15.130770 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127150 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 01:14:15.130770 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127155 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 01:14:15.130770 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127159 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 01:14:15.130770 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127165 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 01:14:15.130770 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127171 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 01:14:15.130770 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127176 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 01:14:15.130770 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127181 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 01:14:15.130770 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127185 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 01:14:15.130770 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127189 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 01:14:15.130770 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127194 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 01:14:15.130770 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127198 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 01:14:15.130770 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127202 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 01:14:15.130770 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127206 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 01:14:15.130770 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127211 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 01:14:15.130770 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127215 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 01:14:15.130770 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127219 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 01:14:15.130770 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127224 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 01:14:15.130770 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127229 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 01:14:15.131261 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127233 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 01:14:15.131261 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127238 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 01:14:15.131261 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127242 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 01:14:15.131261 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127246 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 01:14:15.131261 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127253 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 01:14:15.131261 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127258 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 01:14:15.131261 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127263 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 01:14:15.131261 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127267 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 01:14:15.131261 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127272 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 01:14:15.131261 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127276 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 01:14:15.131261 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127280 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 01:14:15.131261 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127285 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 01:14:15.131261 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127289 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 01:14:15.131261 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127293 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 01:14:15.131261 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127298 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 01:14:15.131261 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127302 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 01:14:15.131261 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127306 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 01:14:15.131261 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127310 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 01:14:15.131261 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127314 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 01:14:15.131261 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127319 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 01:14:15.131261 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127324 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 01:14:15.131775 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127328 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 01:14:15.131775 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127339 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 01:14:15.131775 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127343 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 01:14:15.131775 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127348 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 01:14:15.131775 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127352 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 01:14:15.131775 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127356 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 01:14:15.131775 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127361 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 01:14:15.131775 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127365 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 01:14:15.131775 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127389 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 01:14:15.131775 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127394 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 01:14:15.131775 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.127399 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 01:14:15.131775 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129593 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 01:14:15.131775 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129607 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 01:14:15.131775 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129611 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 01:14:15.131775 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129614 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 01:14:15.131775 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129617 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 01:14:15.131775 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129620 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 01:14:15.131775 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129623 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 01:14:15.131775 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129626 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 01:14:15.132212 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129629 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 01:14:15.132212 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129631 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 01:14:15.132212 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129634 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 01:14:15.132212 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129637 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 01:14:15.132212 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129639 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 01:14:15.132212 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129642 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 01:14:15.132212 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129645 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 01:14:15.132212 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129647 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 01:14:15.132212 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129650 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 01:14:15.132212 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129653 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 01:14:15.132212 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129655 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 01:14:15.132212 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129658 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 01:14:15.132212 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129660 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 01:14:15.132212 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129663 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 01:14:15.132212 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129666 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 01:14:15.132212 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129668 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 01:14:15.132212 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129671 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 01:14:15.132212 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129674 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 01:14:15.132212 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129677 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 01:14:15.132212 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129680 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 01:14:15.132709 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129683 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 01:14:15.132709 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129686 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 01:14:15.132709 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129688 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 01:14:15.132709 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129691 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 01:14:15.132709 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129693 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 01:14:15.132709 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129696 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 01:14:15.132709 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129698 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 01:14:15.132709 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129701 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 01:14:15.132709 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129703 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 01:14:15.132709 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129706 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 01:14:15.132709 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129708 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 01:14:15.132709 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129711 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 01:14:15.132709 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129714 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 01:14:15.132709 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129716 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 01:14:15.132709 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129719 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 01:14:15.132709 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129721 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 01:14:15.132709 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129724 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 01:14:15.132709 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129727 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 01:14:15.132709 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129731 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 01:14:15.133159 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129735 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 01:14:15.133159 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129738 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 01:14:15.133159 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129741 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 01:14:15.133159 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129744 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 01:14:15.133159 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129746 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 01:14:15.133159 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129749 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 01:14:15.133159 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129751 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 01:14:15.133159 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129754 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 01:14:15.133159 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129757 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 01:14:15.133159 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129759 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 01:14:15.133159 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129762 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 01:14:15.133159 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129765 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 01:14:15.133159 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129768 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 01:14:15.133159 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129771 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 01:14:15.133159 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129774 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 01:14:15.133159 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129777 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 01:14:15.133159 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129780 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 01:14:15.133159 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129782 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 01:14:15.133159 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129785 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 01:14:15.133626 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129787 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 01:14:15.133626 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129790 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 01:14:15.133626 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129792 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 01:14:15.133626 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129795 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 01:14:15.133626 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129797 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 01:14:15.133626 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129800 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 01:14:15.133626 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129802 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 01:14:15.133626 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129805 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 01:14:15.133626 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129808 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 01:14:15.133626 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129810 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 01:14:15.133626 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129813 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 01:14:15.133626 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129815 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 01:14:15.133626 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129817 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 01:14:15.133626 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129820 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 01:14:15.133626 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129823 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 02 01:14:15.133626 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129826 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 01:14:15.133626 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129828 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 01:14:15.133626 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129833 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 01:14:15.133626 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129835 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 01:14:15.133626 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129838 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 01:14:15.134098 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129840 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 01:14:15.134098 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129843 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 01:14:15.134098 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129846 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 01:14:15.134098 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129849 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 01:14:15.134098 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129852 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 01:14:15.134098 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129855 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 01:14:15.134098 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129857 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 01:14:15.134098 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129860 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 01:14:15.134098 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129867 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 01:14:15.134098 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129870 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 01:14:15.134098 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129872 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 01:14:15.134098 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129875 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 01:14:15.134098 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129878 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 01:14:15.134098 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129880 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 01:14:15.134098 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129885 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 01:14:15.134098 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129888 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 01:14:15.134098 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129891 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 01:14:15.134098 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129894 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 01:14:15.134098 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129896 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 01:14:15.134564 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.129899 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 01:14:15.134564 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131004 2578 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 01:14:15.134564 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131015 2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 01:14:15.134564 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131023 2578 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 01:14:15.134564 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131028 2578 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 01:14:15.134564 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131033 2578 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 01:14:15.134564 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131036 2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 01:14:15.134564 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131041 2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 01:14:15.134564 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131045 2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 01:14:15.134564 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131048 2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 01:14:15.134564 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131051 2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 01:14:15.134564 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131056 2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 01:14:15.134564 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131059 2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 01:14:15.134564 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131062 2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 01:14:15.134564 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131065 2578 flags.go:64] FLAG: --cgroup-root="" Nov 02 01:14:15.134564 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131068 2578 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 01:14:15.134564 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131072 2578 flags.go:64] FLAG: --client-ca-file="" Nov 02 01:14:15.134564 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131075 2578 flags.go:64] FLAG: --cloud-provider="external" Nov 02 01:14:15.134564 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131078 2578 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 01:14:15.134564 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131082 2578 flags.go:64] FLAG: --cluster-domain="" Nov 02 01:14:15.134564 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131085 2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 01:14:15.134564 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131088 2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 01:14:15.134564 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131091 2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 01:14:15.134564 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131094 2578 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 01:14:15.135137 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131098 2578 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 01:14:15.135137 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131102 2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 01:14:15.135137 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131105 2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 01:14:15.135137 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131108 2578 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 01:14:15.135137 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131111 2578 flags.go:64] FLAG: --contention-profiling="false" Nov 02 01:14:15.135137 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131117 2578 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 01:14:15.135137 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131120 2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 01:14:15.135137 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131123 2578 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 01:14:15.135137 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131126 2578 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 01:14:15.135137 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131131 2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 01:14:15.135137 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131134 2578 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 01:14:15.135137 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131137 2578 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 01:14:15.135137 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131139 2578 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 01:14:15.135137 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131142 2578 flags.go:64] FLAG: --enable-server="true" Nov 02 01:14:15.135137 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131145 2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 01:14:15.135137 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131150 2578 flags.go:64] FLAG: --event-burst="100" Nov 02 01:14:15.135137 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131153 2578 flags.go:64] FLAG: --event-qps="50" Nov 02 01:14:15.135137 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131155 2578 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 01:14:15.135137 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131159 2578 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 01:14:15.135137 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131162 2578 flags.go:64] FLAG: --eviction-hard="" Nov 02 01:14:15.135137 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131167 2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 01:14:15.135137 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131170 2578 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 01:14:15.135137 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131173 2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 01:14:15.135137 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131175 2578 flags.go:64] FLAG: --eviction-soft="" Nov 02 01:14:15.135137 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131179 2578 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 01:14:15.135808 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131182 2578 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 01:14:15.135808 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131185 2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 01:14:15.135808 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131188 2578 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 01:14:15.135808 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131191 2578 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 01:14:15.135808 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131194 2578 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 01:14:15.135808 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131197 2578 flags.go:64] FLAG: --feature-gates="" Nov 02 01:14:15.135808 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131202 2578 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 01:14:15.135808 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131205 2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 01:14:15.135808 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131208 2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 01:14:15.135808 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131211 2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 01:14:15.135808 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131214 2578 flags.go:64] FLAG: --healthz-port="10248" Nov 02 01:14:15.135808 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131217 2578 flags.go:64] FLAG: --help="false" Nov 02 01:14:15.135808 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131220 2578 flags.go:64] FLAG: --hostname-override="ip-10-0-136-130.us-east-2.compute.internal" Nov 02 01:14:15.135808 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131223 2578 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 01:14:15.135808 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131226 2578 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 01:14:15.135808 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131229 2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 01:14:15.135808 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131232 2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 01:14:15.135808 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131235 2578 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 01:14:15.135808 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131238 2578 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 01:14:15.135808 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131241 2578 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 01:14:15.135808 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131244 2578 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 01:14:15.135808 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131246 2578 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 01:14:15.135808 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131249 2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 01:14:15.136362 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131252 2578 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 01:14:15.136362 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131255 2578 flags.go:64] FLAG: --kube-reserved="" Nov 02 01:14:15.136362 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131258 2578 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 01:14:15.136362 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131261 2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 01:14:15.136362 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131264 2578 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 01:14:15.136362 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131269 2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 01:14:15.136362 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131272 2578 flags.go:64] FLAG: --lock-file="" Nov 02 01:14:15.136362 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131275 2578 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 01:14:15.136362 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131279 2578 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 01:14:15.136362 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131282 2578 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 01:14:15.136362 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131288 2578 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 01:14:15.136362 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131291 2578 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 01:14:15.136362 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131294 2578 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 01:14:15.136362 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131297 2578 flags.go:64] FLAG: --logging-format="text" Nov 02 01:14:15.136362 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131300 2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 01:14:15.136362 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131304 2578 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 01:14:15.136362 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131307 2578 flags.go:64] FLAG: --manifest-url="" Nov 02 01:14:15.136362 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131310 2578 flags.go:64] FLAG: --manifest-url-header="" Nov 02 01:14:15.136362 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131314 2578 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 01:14:15.136362 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131317 2578 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 01:14:15.136362 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131321 2578 flags.go:64] FLAG: --max-pods="110" Nov 02 01:14:15.136362 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131324 2578 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 01:14:15.136362 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131327 2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 01:14:15.136362 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131330 2578 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 01:14:15.136362 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131333 2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 01:14:15.136966 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131336 2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 01:14:15.136966 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131339 2578 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 01:14:15.136966 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131342 2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 01:14:15.136966 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131349 2578 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 01:14:15.136966 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131352 2578 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 01:14:15.136966 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131355 2578 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 01:14:15.136966 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131358 2578 flags.go:64] FLAG: --pod-cidr="" Nov 02 01:14:15.136966 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131361 2578 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 01:14:15.136966 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131364 2578 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 01:14:15.136966 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131366 2578 flags.go:64] FLAG: --pods-per-core="0" Nov 02 01:14:15.136966 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131386 2578 flags.go:64] FLAG: --port="10250" Nov 02 01:14:15.136966 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131391 2578 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 01:14:15.136966 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131395 2578 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-057ef1ed1b55324a5" Nov 02 01:14:15.136966 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131399 2578 flags.go:64] FLAG: --qos-reserved="" Nov 02 01:14:15.136966 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131402 2578 flags.go:64] FLAG: --read-only-port="10255" Nov 02 01:14:15.136966 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131405 2578 flags.go:64] FLAG: --register-node="true" Nov 02 01:14:15.136966 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131408 2578 flags.go:64] FLAG: --register-with-taints="" Nov 02 01:14:15.136966 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131415 2578 flags.go:64] FLAG: --registry-burst="10" Nov 02 01:14:15.136966 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131418 2578 flags.go:64] FLAG: --registry-qps="5" Nov 02 01:14:15.136966 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131421 2578 flags.go:64] FLAG: --reserved-cpus="" Nov 02 01:14:15.136966 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131424 2578 flags.go:64] FLAG: --reserved-memory="" Nov 02 01:14:15.136966 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131427 2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 01:14:15.136966 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131430 2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 01:14:15.136966 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131433 2578 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 01:14:15.136966 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131436 2578 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 01:14:15.136966 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131439 2578 flags.go:64] FLAG: --runonce="false" Nov 02 01:14:15.137591 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131442 2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 01:14:15.137591 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131445 2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 01:14:15.137591 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131448 2578 flags.go:64] FLAG: --seccomp-default="false" Nov 02 01:14:15.137591 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131451 2578 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 01:14:15.137591 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131454 2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 01:14:15.137591 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131457 2578 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 01:14:15.137591 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131460 2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 01:14:15.137591 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131463 2578 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 01:14:15.137591 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131466 2578 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 01:14:15.137591 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131469 2578 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 01:14:15.137591 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131472 2578 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 01:14:15.137591 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131475 2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 01:14:15.137591 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131478 2578 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 01:14:15.137591 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131481 2578 flags.go:64] FLAG: --system-cgroups="" Nov 02 01:14:15.137591 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131483 2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 01:14:15.137591 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131490 2578 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 01:14:15.137591 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131493 2578 flags.go:64] FLAG: --tls-cert-file="" Nov 02 01:14:15.137591 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131496 2578 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 01:14:15.137591 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131500 2578 flags.go:64] FLAG: --tls-min-version="" Nov 02 01:14:15.137591 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131503 2578 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 01:14:15.137591 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131507 2578 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 01:14:15.137591 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131511 2578 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 01:14:15.137591 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131514 2578 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 01:14:15.137591 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131518 2578 flags.go:64] FLAG: --v="2" Nov 02 01:14:15.137591 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131523 2578 flags.go:64] FLAG: --version="false" Nov 02 01:14:15.138222 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131527 2578 flags.go:64] FLAG: --vmodule="" Nov 02 01:14:15.138222 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131531 2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 01:14:15.138222 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.131534 2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 01:14:15.138222 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131699 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 01:14:15.138222 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131703 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 01:14:15.138222 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131706 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 01:14:15.138222 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131709 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 01:14:15.138222 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131711 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 01:14:15.138222 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131714 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 01:14:15.138222 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131717 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 01:14:15.138222 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131721 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 01:14:15.138222 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131725 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 01:14:15.138222 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131728 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 01:14:15.138222 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131731 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 01:14:15.138222 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131734 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 01:14:15.138222 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131736 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 01:14:15.138222 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131739 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 01:14:15.138222 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131741 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 01:14:15.138222 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131744 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 01:14:15.138222 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131747 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 01:14:15.139101 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131749 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 01:14:15.139101 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131751 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 01:14:15.139101 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131754 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 01:14:15.139101 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131756 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 01:14:15.139101 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131759 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 01:14:15.139101 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131761 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 01:14:15.139101 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131764 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 01:14:15.139101 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131766 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 01:14:15.139101 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131771 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 01:14:15.139101 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131774 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 01:14:15.139101 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131778 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 01:14:15.139101 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131781 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 01:14:15.139101 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131783 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 01:14:15.139101 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131786 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 01:14:15.139101 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131789 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 01:14:15.139101 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131791 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 01:14:15.139101 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131794 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 01:14:15.139101 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131796 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 01:14:15.139101 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131799 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 01:14:15.139101 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131802 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 01:14:15.139956 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131804 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 01:14:15.139956 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131807 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 01:14:15.139956 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131809 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 01:14:15.139956 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131812 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 01:14:15.139956 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131814 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 01:14:15.139956 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131817 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 01:14:15.139956 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131820 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 01:14:15.139956 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131822 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 01:14:15.139956 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131824 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 01:14:15.139956 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131827 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 01:14:15.139956 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131829 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 01:14:15.139956 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131832 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 01:14:15.139956 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131834 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 02 01:14:15.139956 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131836 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 01:14:15.139956 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131839 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 01:14:15.139956 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131841 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 01:14:15.139956 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131844 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 01:14:15.139956 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131846 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 01:14:15.139956 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131849 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 01:14:15.140755 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131851 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 01:14:15.140755 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131855 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 01:14:15.140755 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131858 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 01:14:15.140755 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131861 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 01:14:15.140755 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131864 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 01:14:15.140755 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131866 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 01:14:15.140755 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131869 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 01:14:15.140755 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131871 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 01:14:15.140755 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131873 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 01:14:15.140755 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131876 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 01:14:15.140755 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131878 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 01:14:15.140755 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131881 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 01:14:15.140755 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131883 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 01:14:15.140755 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131886 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 01:14:15.140755 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131889 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 01:14:15.140755 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131893 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 01:14:15.140755 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131896 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 01:14:15.140755 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131899 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 01:14:15.140755 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131901 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 01:14:15.141646 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131904 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 01:14:15.141646 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131906 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 01:14:15.141646 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131909 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 01:14:15.141646 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131911 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 01:14:15.141646 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131914 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 01:14:15.141646 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131916 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 01:14:15.141646 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131919 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 01:14:15.141646 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131921 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 01:14:15.141646 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131923 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 01:14:15.141646 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131926 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 01:14:15.141646 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131928 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 01:14:15.141646 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131931 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 01:14:15.141646 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131933 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 01:14:15.141646 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131936 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 01:14:15.141646 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131939 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 01:14:15.141646 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131942 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 01:14:15.141646 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131945 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 01:14:15.141646 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131948 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 01:14:15.141646 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131950 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 01:14:15.141646 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131952 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 01:14:15.142545 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131955 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 01:14:15.142545 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131957 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 01:14:15.142545 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131960 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 01:14:15.142545 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131962 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 01:14:15.142545 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131965 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 01:14:15.142545 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131967 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 01:14:15.142545 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131970 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 01:14:15.142545 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131972 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 01:14:15.142545 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131974 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 01:14:15.142545 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131977 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 01:14:15.142545 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.131979 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 01:14:15.142545 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.132010 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 01:14:15.142545 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132059 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 01:14:15.142545 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132063 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 01:14:15.142545 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132067 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 01:14:15.142545 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132070 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 01:14:15.143155 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132073 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 01:14:15.143155 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132076 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 01:14:15.143155 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132078 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 01:14:15.143155 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132080 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 01:14:15.143155 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132083 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 01:14:15.143155 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132085 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 01:14:15.143155 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132088 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 01:14:15.143155 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132091 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 01:14:15.143155 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132093 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 01:14:15.143155 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132096 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 01:14:15.143155 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132101 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 01:14:15.143155 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132103 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 01:14:15.143155 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132106 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 01:14:15.143155 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132109 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 01:14:15.143155 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132112 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 01:14:15.143155 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132114 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 01:14:15.143155 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132116 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 01:14:15.143155 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132119 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 01:14:15.143155 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132122 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 01:14:15.143155 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132124 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 01:14:15.143809 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132127 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 01:14:15.143809 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132129 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 01:14:15.143809 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132132 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 01:14:15.143809 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132134 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 01:14:15.143809 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132136 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 01:14:15.143809 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132139 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 01:14:15.143809 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132141 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 01:14:15.143809 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132143 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 01:14:15.143809 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132146 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 02 01:14:15.143809 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132148 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 01:14:15.143809 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132151 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 01:14:15.143809 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132153 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 01:14:15.143809 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132156 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 01:14:15.143809 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132158 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 01:14:15.143809 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132161 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 01:14:15.143809 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132163 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 01:14:15.143809 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132166 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 01:14:15.143809 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132168 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 01:14:15.143809 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132171 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 01:14:15.144575 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132173 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 01:14:15.144575 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132176 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 01:14:15.144575 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132178 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 01:14:15.144575 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132183 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 01:14:15.144575 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132185 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 01:14:15.144575 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132188 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 01:14:15.144575 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132190 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 01:14:15.144575 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132194 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 01:14:15.144575 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132196 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 01:14:15.144575 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132199 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 01:14:15.144575 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132203 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 01:14:15.144575 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132206 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 01:14:15.144575 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132209 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 01:14:15.144575 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132211 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 01:14:15.144575 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132214 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 01:14:15.144575 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132217 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 01:14:15.144575 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132219 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 01:14:15.144575 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132222 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 01:14:15.144575 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132225 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 01:14:15.144575 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132227 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 01:14:15.145213 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132230 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 01:14:15.145213 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132232 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 01:14:15.145213 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132235 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 01:14:15.145213 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132237 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 01:14:15.145213 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132240 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 01:14:15.145213 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132242 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 01:14:15.145213 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132244 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 01:14:15.145213 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132247 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 01:14:15.145213 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132249 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 01:14:15.145213 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132253 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 01:14:15.145213 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132256 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 01:14:15.145213 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132258 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 01:14:15.145213 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132261 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 01:14:15.145213 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132264 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 01:14:15.145213 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132266 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 01:14:15.145213 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132269 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 01:14:15.145213 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132272 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 01:14:15.145213 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132274 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 01:14:15.145213 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132277 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 01:14:15.145213 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132279 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 01:14:15.145710 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132283 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 01:14:15.145710 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132285 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 01:14:15.145710 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132288 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 01:14:15.145710 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132290 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 01:14:15.145710 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132292 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 01:14:15.145710 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132295 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 01:14:15.145710 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132297 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 01:14:15.145710 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132300 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 01:14:15.145710 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132303 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 01:14:15.145710 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132305 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 01:14:15.145710 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132308 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 01:14:15.145710 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132310 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 01:14:15.145710 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132313 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 01:14:15.145710 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132316 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 01:14:15.145710 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132320 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 01:14:15.145710 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132324 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 01:14:15.145710 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132327 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 01:14:15.145710 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132330 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 01:14:15.145710 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132333 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 01:14:15.146347 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132336 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 01:14:15.146347 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132338 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 01:14:15.146347 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132342 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 01:14:15.146347 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.132344 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 01:14:15.146347 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.140845 2578 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 01:14:15.146347 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.140863 2578 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 01:14:15.146347 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.140887 2578 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 01:14:15.146347 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.140894 2578 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 01:14:15.146347 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.140981 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 01:14:15.146347 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.140990 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 01:14:15.146347 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.140996 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 01:14:15.146347 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141002 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 01:14:15.146347 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141006 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 01:14:15.146347 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141012 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 01:14:15.146347 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141016 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 01:14:15.146347 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141020 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 01:14:15.146347 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141024 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 01:14:15.146347 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141029 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 01:14:15.146347 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141033 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 01:14:15.146347 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141038 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 01:14:15.146849 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141042 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 01:14:15.146849 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141046 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 01:14:15.146849 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141050 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 01:14:15.146849 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141054 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 01:14:15.146849 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141059 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 01:14:15.146849 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141063 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 01:14:15.146849 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141067 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 01:14:15.146849 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141071 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 01:14:15.146849 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141075 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 01:14:15.146849 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141080 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 01:14:15.146849 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141084 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 01:14:15.146849 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141088 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 01:14:15.146849 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141093 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 01:14:15.146849 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141097 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 01:14:15.146849 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141101 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 01:14:15.146849 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141105 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 01:14:15.146849 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141109 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 01:14:15.146849 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141114 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 01:14:15.146849 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141119 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 01:14:15.147355 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141123 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 01:14:15.147355 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141128 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 01:14:15.147355 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141132 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 01:14:15.147355 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141136 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 01:14:15.147355 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141140 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 01:14:15.147355 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141144 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 01:14:15.147355 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141149 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 01:14:15.147355 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141154 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 01:14:15.147355 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141158 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 01:14:15.147355 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141163 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 01:14:15.147355 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141168 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 01:14:15.147355 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141172 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 01:14:15.147355 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141176 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 01:14:15.147355 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141180 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 01:14:15.147355 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141184 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 01:14:15.147355 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141188 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 01:14:15.147355 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141193 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 01:14:15.147355 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141197 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 01:14:15.147355 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141201 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 01:14:15.147355 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141207 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 01:14:15.147915 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141214 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 01:14:15.147915 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141219 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 01:14:15.147915 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141225 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 01:14:15.147915 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141229 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 01:14:15.147915 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141234 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 01:14:15.147915 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141238 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 01:14:15.147915 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141242 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 01:14:15.147915 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141246 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 01:14:15.147915 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141251 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 01:14:15.147915 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141255 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 01:14:15.147915 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141260 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 01:14:15.147915 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141265 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 01:14:15.147915 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141269 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 02 01:14:15.147915 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141273 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 01:14:15.147915 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141277 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 01:14:15.147915 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141281 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 01:14:15.147915 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141285 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 01:14:15.147915 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141289 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 01:14:15.147915 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141294 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 01:14:15.147915 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141298 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 01:14:15.148543 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141304 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 01:14:15.148543 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141308 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 01:14:15.148543 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141312 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 01:14:15.148543 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141316 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 01:14:15.148543 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141321 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 01:14:15.148543 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141326 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 01:14:15.148543 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141331 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 01:14:15.148543 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141335 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 01:14:15.148543 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141339 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 01:14:15.148543 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141343 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 01:14:15.148543 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141347 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 01:14:15.148543 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141351 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 01:14:15.148543 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141356 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 01:14:15.148543 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141362 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 01:14:15.148543 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141367 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 01:14:15.148543 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141390 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 01:14:15.148543 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141394 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 01:14:15.148543 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141399 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 01:14:15.148543 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141403 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 01:14:15.149025 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141407 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 01:14:15.149025 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141411 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 01:14:15.149025 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141415 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 01:14:15.149025 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141420 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 01:14:15.149025 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141424 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 01:14:15.149025 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141428 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 01:14:15.149025 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141433 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 01:14:15.149025 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141437 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 01:14:15.149025 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141441 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 01:14:15.149025 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141445 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 01:14:15.149025 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141450 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 01:14:15.149025 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141455 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 01:14:15.149025 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141460 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 01:14:15.149025 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141465 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 01:14:15.149025 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141470 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 01:14:15.149025 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141475 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 01:14:15.149479 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.141518 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 01:14:15.149479 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141749 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 01:14:15.149479 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141757 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 01:14:15.149479 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141762 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 01:14:15.149479 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141768 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 01:14:15.149479 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141772 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 01:14:15.149479 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141777 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 01:14:15.149479 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141781 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 01:14:15.149479 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141785 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 01:14:15.149479 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141789 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 01:14:15.149479 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141793 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 01:14:15.149479 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141797 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 01:14:15.149479 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141801 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 01:14:15.149479 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141805 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 01:14:15.149479 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141810 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 01:14:15.149851 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141814 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 01:14:15.149851 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141819 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 01:14:15.149851 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141825 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 01:14:15.149851 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141831 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 01:14:15.149851 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141836 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 01:14:15.149851 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141841 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 01:14:15.149851 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141845 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 01:14:15.149851 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141849 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 01:14:15.149851 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141853 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 01:14:15.149851 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141857 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 01:14:15.149851 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141861 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 01:14:15.149851 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141865 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 01:14:15.149851 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141869 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 01:14:15.149851 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141874 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 01:14:15.149851 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141878 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 01:14:15.149851 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141882 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 01:14:15.149851 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141887 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 01:14:15.149851 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141892 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 01:14:15.149851 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141896 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 01:14:15.150354 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141900 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 01:14:15.150354 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141904 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 01:14:15.150354 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141908 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 01:14:15.150354 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141912 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 01:14:15.150354 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141916 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 01:14:15.150354 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141920 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 01:14:15.150354 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141924 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 02 01:14:15.150354 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141928 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 01:14:15.150354 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141932 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 01:14:15.150354 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141936 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 01:14:15.150354 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141940 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 01:14:15.150354 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141945 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 01:14:15.150354 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141948 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 01:14:15.150354 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141952 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 01:14:15.150354 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141957 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 01:14:15.150354 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141961 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 01:14:15.150354 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141965 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 01:14:15.150354 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141969 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 01:14:15.150354 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141974 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 01:14:15.150354 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141978 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 01:14:15.150864 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141983 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 01:14:15.150864 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141986 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 01:14:15.150864 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141990 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 01:14:15.150864 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141994 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 01:14:15.150864 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.141999 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 01:14:15.150864 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142003 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 01:14:15.150864 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142007 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 01:14:15.150864 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142011 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 01:14:15.150864 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142015 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 01:14:15.150864 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142019 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 01:14:15.150864 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142025 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 01:14:15.150864 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142029 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 01:14:15.150864 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142033 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 01:14:15.150864 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142037 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 01:14:15.150864 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142041 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 01:14:15.150864 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142045 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 01:14:15.150864 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142049 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 01:14:15.150864 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142053 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 01:14:15.150864 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142057 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 01:14:15.150864 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142061 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 01:14:15.151405 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142065 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 01:14:15.151405 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142070 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 01:14:15.151405 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142074 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 01:14:15.151405 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142078 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 01:14:15.151405 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142082 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 01:14:15.151405 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142085 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 01:14:15.151405 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142090 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 01:14:15.151405 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142094 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 01:14:15.151405 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142098 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 01:14:15.151405 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142102 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 01:14:15.151405 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142107 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 01:14:15.151405 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142111 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 01:14:15.151405 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142115 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 01:14:15.151405 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142121 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 01:14:15.151405 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142126 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 01:14:15.151405 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142131 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 01:14:15.151405 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142136 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 01:14:15.151405 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142140 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 01:14:15.151405 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142144 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 01:14:15.151862 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142149 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 01:14:15.151862 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142154 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 01:14:15.151862 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142158 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 01:14:15.151862 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142162 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 01:14:15.151862 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142167 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 01:14:15.151862 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142171 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 01:14:15.151862 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142175 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 01:14:15.151862 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142180 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 01:14:15.151862 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142184 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 01:14:15.151862 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142188 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 01:14:15.151862 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142192 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 01:14:15.151862 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142196 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 01:14:15.151862 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142200 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 01:14:15.151862 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142204 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 01:14:15.151862 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.142242 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 01:14:15.151862 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142295 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 01:14:15.152257 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142302 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 01:14:15.152257 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142306 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 01:14:15.152257 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142311 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 01:14:15.152257 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142315 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 01:14:15.152257 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142319 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 01:14:15.152257 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142323 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 01:14:15.152257 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142329 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 01:14:15.152257 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142333 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 01:14:15.152257 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142337 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 01:14:15.152257 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142341 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 01:14:15.152257 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142345 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 01:14:15.152257 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142349 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 01:14:15.152257 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142354 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 01:14:15.152257 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142358 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 01:14:15.152257 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142362 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 01:14:15.152257 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142366 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 01:14:15.152257 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142389 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 01:14:15.152257 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142394 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 01:14:15.152257 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142398 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 01:14:15.152722 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142402 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 01:14:15.152722 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142406 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 01:14:15.152722 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142417 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 01:14:15.152722 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142422 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 01:14:15.152722 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142426 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 01:14:15.152722 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142430 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 01:14:15.152722 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142436 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 01:14:15.152722 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142443 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 01:14:15.152722 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142448 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 01:14:15.152722 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142453 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 01:14:15.152722 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142457 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 01:14:15.152722 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142461 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 01:14:15.152722 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142465 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 01:14:15.152722 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142469 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 01:14:15.152722 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142473 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 01:14:15.152722 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142480 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 01:14:15.152722 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142484 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 01:14:15.152722 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142489 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 01:14:15.152722 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142493 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 01:14:15.153463 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142498 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 01:14:15.153463 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142502 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 01:14:15.153463 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142507 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 01:14:15.153463 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142511 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 01:14:15.153463 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142516 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 01:14:15.153463 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142520 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 01:14:15.153463 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142524 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 01:14:15.153463 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142529 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 01:14:15.153463 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142533 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 01:14:15.153463 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142538 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 01:14:15.153463 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142542 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 01:14:15.153463 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142546 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 01:14:15.153463 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142550 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 01:14:15.153463 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142554 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 01:14:15.153463 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142558 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 01:14:15.153463 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142563 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 01:14:15.153463 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142567 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 01:14:15.153463 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142572 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 01:14:15.153463 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142576 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 01:14:15.154202 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142580 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 01:14:15.154202 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142585 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 01:14:15.154202 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142589 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 01:14:15.154202 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142593 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 01:14:15.154202 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142600 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 01:14:15.154202 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142606 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 01:14:15.154202 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142610 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 01:14:15.154202 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142615 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 01:14:15.154202 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142619 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 01:14:15.154202 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142625 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 01:14:15.154202 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142629 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 01:14:15.154202 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142633 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 01:14:15.154202 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142638 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 01:14:15.154202 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142643 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 01:14:15.154202 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142647 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 01:14:15.154202 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142652 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 01:14:15.154202 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142656 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 01:14:15.154202 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142660 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 01:14:15.154202 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142664 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 01:14:15.155038 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142669 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 01:14:15.155038 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142673 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 01:14:15.155038 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142677 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 01:14:15.155038 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142682 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 01:14:15.155038 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142686 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 01:14:15.155038 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142690 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 01:14:15.155038 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142694 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 01:14:15.155038 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142698 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 01:14:15.155038 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142702 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 02 01:14:15.155038 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142707 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 01:14:15.155038 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142711 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 01:14:15.155038 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142728 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 01:14:15.155038 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142733 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 01:14:15.155038 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142739 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 01:14:15.155038 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142743 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 01:14:15.155038 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142747 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 01:14:15.155038 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142751 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 01:14:15.155038 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142755 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 01:14:15.155038 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142759 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 01:14:15.155038 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142763 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 01:14:15.155943 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142769 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 01:14:15.155943 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142774 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 01:14:15.155943 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142779 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 01:14:15.155943 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142784 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 01:14:15.155943 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142788 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 01:14:15.155943 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142792 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 01:14:15.155943 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142798 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 01:14:15.155943 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142803 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 01:14:15.155943 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:15.142807 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 01:14:15.155943 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.142892 2578 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 01:14:15.155943 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.148055 2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 02 01:14:15.155943 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.149228 2578 server.go:1014] "Starting client certificate rotation" Nov 02 01:14:15.155943 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.149326 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 01:14:15.155943 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.149362 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 01:14:15.155943 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.153538 2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 01:14:15.156320 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.155969 2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 01:14:15.172315 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.172295 2578 log.go:25] "Validated CRI v1 runtime API" Nov 02 01:14:15.176401 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.176367 2578 log.go:25] "Validated CRI v1 image API" Nov 02 01:14:15.177183 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.177158 2578 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 01:14:15.179904 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.179881 2578 fs.go:135] Filesystem UUIDs: map[21769c05-33cb-460b-a1d9-070bf1bcfad1:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 b22bfa15-235d-4db4-869a-1045cba4b495:/dev/nvme0n1p3] Nov 02 01:14:15.179964 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.179902 2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 02 01:14:15.185352 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.185331 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 01:14:15.185801 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.185688 2578 manager.go:217] Machine: {Timestamp:2026-11-02 01:14:15.184168088 +0000 UTC m=+0.390460742 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3132188 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec270def98fc1760c6c00ee0312e9ccb SystemUUID:ec270def-98fc-1760-c6c0-0ee0312e9ccb BootID:6b66e05d-81a8-4da7-b7be-4c59ed138df8 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e6:7a:af:b1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e6:7a:af:b1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:46:79:53:ff:38:00 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 01:14:15.186449 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.186438 2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 01:14:15.186550 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.186538 2578 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 01:14:15.187786 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.187764 2578 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 01:14:15.187925 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.187788 2578 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-130.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 01:14:15.187964 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.187931 2578 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 01:14:15.187964 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.187939 2578 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 01:14:15.187964 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.187953 2578 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 01:14:15.187964 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.187964 2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 01:14:15.189172 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.189160 2578 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 01:14:15.193940 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.193927 2578 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 01:14:15.194067 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.194057 2578 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 01:14:15.194104 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.194092 2578 kubelet.go:493] "Attempting to sync node with API server" Nov 02 01:14:15.194159 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.194105 2578 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 01:14:15.194159 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.194126 2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 01:14:15.194159 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.194139 2578 kubelet.go:405] "Adding apiserver pod source" Nov 02 01:14:15.194159 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.194148 2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 01:14:15.194539 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.194523 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 01:14:15.197083 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.197049 2578 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 01:14:15.204996 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.204981 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 01:14:15.205970 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.205956 2578 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 01:14:15.206010 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.205988 2578 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 01:14:15.206146 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.206133 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 01:14:15.206190 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.206157 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 01:14:15.206190 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.206166 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 01:14:15.206190 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.206175 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 01:14:15.206190 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.206182 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 01:14:15.206190 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.206190 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 01:14:15.206839 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.206828 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 01:14:15.206839 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.206840 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 01:14:15.206892 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.206847 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 01:14:15.206892 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.206858 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 01:14:15.206892 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.206872 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 01:14:15.206892 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.206880 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 01:14:15.207955 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.207939 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-97njz" Nov 02 01:14:15.208442 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.208431 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 01:14:15.208473 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.208444 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 01:14:15.214796 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.214771 2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-130.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 02 01:14:15.215158 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.215144 2578 server.go:1263] "Started kubelet" Nov 02 01:14:15.215272 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.215196 2578 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 01:14:15.215354 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.215297 2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 01:14:15.215410 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.215397 2578 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 01:14:15.216083 ip-10-0-136-130 systemd[1]: Started Kubernetes Kubelet. Nov 02 01:14:15.216872 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.216857 2578 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 01:14:15.217849 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.217827 2578 server.go:317] "Adding debug handlers to kubelet server" Nov 02 01:14:15.221559 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:15.221541 2578 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 02 01:14:15.221910 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.221893 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-97njz" Nov 02 01:14:15.225140 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.225117 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 01:14:15.225227 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.225132 2578 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 01:14:15.225277 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:15.225225 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-130.us-east-2.compute.internal\" not found" Nov 02 01:14:15.225924 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.225907 2578 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 01:14:15.225924 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.225925 2578 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 01:14:15.226063 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.225910 2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 01:14:15.226063 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.225953 2578 factory.go:55] Registering systemd factory Nov 02 01:14:15.226063 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.225968 2578 factory.go:223] Registration of the systemd container factory successfully Nov 02 01:14:15.226063 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.226008 2578 reconstruct.go:97] "Volume reconstruction finished" Nov 02 01:14:15.226063 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.226017 2578 reconciler.go:29] "Reconciler: start to sync state" Nov 02 01:14:15.226248 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.226183 2578 factory.go:153] Registering CRI-O factory Nov 02 01:14:15.226248 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.226198 2578 factory.go:223] Registration of the crio container factory successfully Nov 02 01:14:15.226248 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.226246 2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 01:14:15.226361 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.226265 2578 factory.go:103] Registering Raw factory Nov 02 01:14:15.226361 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.226279 2578 manager.go:1199] Started watching for new ooms in manager Nov 02 01:14:15.228523 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.227546 2578 manager.go:319] Starting recovery of all containers Nov 02 01:14:15.228523 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:15.227857 2578 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-130.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 02 01:14:15.228523 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:15.228474 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-130.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 02 01:14:15.228690 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:15.228578 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 02 01:14:15.229175 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:15.227904 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-130.us-east-2.compute.internal.18e415605c9c7cab default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-130.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-130.us-east-2.compute.internal,},FirstTimestamp:2026-11-02 01:14:15.215103147 +0000 UTC m=+0.421395799,LastTimestamp:2026-11-02 01:14:15.215103147 +0000 UTC m=+0.421395799,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-130.us-east-2.compute.internal,}" Nov 02 01:14:15.232841 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:15.232811 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 02 01:14:15.239325 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.239303 2578 manager.go:324] Recovery completed Nov 02 01:14:15.244513 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.244496 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 01:14:15.246335 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.246316 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-130.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 01:14:15.246426 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.246349 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-130.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 01:14:15.246426 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.246361 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-130.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 02 01:14:15.246970 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.246952 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 01:14:15.247030 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.246956 2578 cpu_manager.go:226] "Starting" policy="none" Nov 02 01:14:15.247030 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.247011 2578 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 01:14:15.247030 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.247028 2578 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 01:14:15.247936 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.247916 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-130.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 01:14:15.247998 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.247955 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-130.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 01:14:15.247998 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.247972 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-130.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 02 01:14:15.249255 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.249236 2578 policy_none.go:50] "Start" Nov 02 01:14:15.249982 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.249967 2578 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 01:14:15.250044 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.249993 2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 01:14:15.252144 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.252132 2578 policy_none.go:44] "Start" Nov 02 01:14:15.293136 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.292962 2578 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 01:14:15.297634 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.297616 2578 manager.go:341] "Starting Device Plugin manager" Nov 02 01:14:15.298295 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:15.298282 2578 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 01:14:15.298295 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.298296 2578 server.go:87] "Starting device plugin registration server" Nov 02 01:14:15.298561 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.298550 2578 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 01:14:15.298637 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.298563 2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 01:14:15.298689 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.298650 2578 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 01:14:15.298749 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.298737 2578 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 01:14:15.298749 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.298748 2578 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 01:14:15.299319 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:15.299302 2578 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 01:14:15.299403 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:15.299345 2578 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-130.us-east-2.compute.internal\" not found" Nov 02 01:14:15.360709 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.360666 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 01:14:15.362810 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.362021 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 01:14:15.362810 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.362474 2578 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 01:14:15.362810 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.362503 2578 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 01:14:15.362810 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:15.362539 2578 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 02 01:14:15.366331 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.366309 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 01:14:15.399458 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.399386 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 01:14:15.400339 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.400316 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-130.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 01:14:15.400451 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.400352 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-130.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 01:14:15.400451 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.400368 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-130.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 02 01:14:15.400451 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.400428 2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-130.us-east-2.compute.internal" Nov 02 01:14:15.408846 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.408832 2578 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-130.us-east-2.compute.internal" Nov 02 01:14:15.408907 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:15.408857 2578 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-130.us-east-2.compute.internal\": node \"ip-10-0-136-130.us-east-2.compute.internal\" not found" Nov 02 01:14:15.423752 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:15.423723 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-130.us-east-2.compute.internal\" not found" Nov 02 01:14:15.462983 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.462958 2578 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-130.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-130.us-east-2.compute.internal"] Nov 02 01:14:15.463100 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.463017 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 01:14:15.463955 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.463929 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-130.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 01:14:15.464067 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.463964 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-130.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 01:14:15.464067 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.463980 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-130.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 02 01:14:15.466190 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.466173 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 01:14:15.466361 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.466347 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-130.us-east-2.compute.internal" Nov 02 01:14:15.466437 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.466393 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 01:14:15.466942 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.466920 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-130.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 01:14:15.467040 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.466951 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-130.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 01:14:15.467040 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.466966 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-130.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 02 01:14:15.467040 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.466979 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-130.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 01:14:15.467040 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.467002 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-130.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 01:14:15.467040 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.467016 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-130.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 02 01:14:15.469117 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.469101 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-130.us-east-2.compute.internal" Nov 02 01:14:15.469164 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.469129 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 01:14:15.470111 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.470090 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-130.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 01:14:15.470215 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.470118 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-130.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 01:14:15.470215 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.470128 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-130.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 02 01:14:15.497226 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:15.497207 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-130.us-east-2.compute.internal\" not found" node="ip-10-0-136-130.us-east-2.compute.internal" Nov 02 01:14:15.499902 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:15.499885 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-130.us-east-2.compute.internal\" not found" node="ip-10-0-136-130.us-east-2.compute.internal" Nov 02 01:14:15.524570 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:15.524548 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-130.us-east-2.compute.internal\" not found" Nov 02 01:14:15.527817 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.527799 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dabfce80149e41ad659d129c965945bf-config\") pod \"kube-apiserver-proxy-ip-10-0-136-130.us-east-2.compute.internal\" (UID: \"dabfce80149e41ad659d129c965945bf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-130.us-east-2.compute.internal" Nov 02 01:14:15.527880 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.527825 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b96c82052e4f897819ee4482b034eaf4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-130.us-east-2.compute.internal\" (UID: \"b96c82052e4f897819ee4482b034eaf4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-130.us-east-2.compute.internal" Nov 02 01:14:15.527916 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.527873 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b96c82052e4f897819ee4482b034eaf4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-130.us-east-2.compute.internal\" (UID: \"b96c82052e4f897819ee4482b034eaf4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-130.us-east-2.compute.internal" Nov 02 01:14:15.625668 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:15.625632 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-130.us-east-2.compute.internal\" not found" Nov 02 01:14:15.628695 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.628680 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dabfce80149e41ad659d129c965945bf-config\") pod \"kube-apiserver-proxy-ip-10-0-136-130.us-east-2.compute.internal\" (UID: \"dabfce80149e41ad659d129c965945bf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-130.us-east-2.compute.internal" Nov 02 01:14:15.628755 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.628705 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b96c82052e4f897819ee4482b034eaf4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-130.us-east-2.compute.internal\" (UID: \"b96c82052e4f897819ee4482b034eaf4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-130.us-east-2.compute.internal" Nov 02 01:14:15.628755 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.628723 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b96c82052e4f897819ee4482b034eaf4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-130.us-east-2.compute.internal\" (UID: \"b96c82052e4f897819ee4482b034eaf4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-130.us-east-2.compute.internal" Nov 02 01:14:15.628755 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.628749 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b96c82052e4f897819ee4482b034eaf4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-130.us-east-2.compute.internal\" (UID: \"b96c82052e4f897819ee4482b034eaf4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-130.us-east-2.compute.internal" Nov 02 01:14:15.628870 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.628785 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dabfce80149e41ad659d129c965945bf-config\") pod \"kube-apiserver-proxy-ip-10-0-136-130.us-east-2.compute.internal\" (UID: \"dabfce80149e41ad659d129c965945bf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-130.us-east-2.compute.internal" Nov 02 01:14:15.628870 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.628791 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b96c82052e4f897819ee4482b034eaf4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-130.us-east-2.compute.internal\" (UID: \"b96c82052e4f897819ee4482b034eaf4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-130.us-east-2.compute.internal" Nov 02 01:14:15.726645 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:15.726560 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-130.us-east-2.compute.internal\" not found" Nov 02 01:14:15.800904 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.800881 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-130.us-east-2.compute.internal" Nov 02 01:14:15.803045 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:15.803024 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-130.us-east-2.compute.internal" Nov 02 01:14:15.827458 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:15.827434 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-130.us-east-2.compute.internal\" not found" Nov 02 01:14:15.928024 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:15.927976 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-130.us-east-2.compute.internal\" not found" Nov 02 01:14:16.028539 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:16.028442 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-130.us-east-2.compute.internal\" not found" Nov 02 01:14:16.129091 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:16.129053 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-130.us-east-2.compute.internal\" not found" Nov 02 01:14:16.149360 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:16.149339 2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 02 01:14:16.149523 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:16.149505 2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 01:14:16.229960 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:16.229935 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-130.us-east-2.compute.internal\" not found" Nov 02 01:14:16.231048 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:16.231008 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 01:09:15 +0000 UTC" deadline="2028-04-01 01:04:59.852469024 +0000 UTC" Nov 02 01:14:16.231094 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:16.231048 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12383h50m43.621423588s" Nov 02 01:14:16.330932 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:16.330859 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-130.us-east-2.compute.internal\" not found" Nov 02 01:14:16.431701 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:16.431657 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-130.us-east-2.compute.internal\" not found" Nov 02 01:14:16.467993 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:16.467973 2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 01:14:16.477421 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:16.477391 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb96c82052e4f897819ee4482b034eaf4.slice/crio-3d2bf19f263abe4ecec40ed8a73450e0a6ece52303e4d7e99934da5b998f2b5b WatchSource:0}: Error finding container 3d2bf19f263abe4ecec40ed8a73450e0a6ece52303e4d7e99934da5b998f2b5b: Status 404 returned error can't find the container with id 3d2bf19f263abe4ecec40ed8a73450e0a6ece52303e4d7e99934da5b998f2b5b Nov 02 01:14:16.477986 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:16.477966 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddabfce80149e41ad659d129c965945bf.slice/crio-2e90faf8dd799960b8070f2406807a3587d50ac8e1a0a04ddb00976f65a9151c WatchSource:0}: Error finding container 2e90faf8dd799960b8070f2406807a3587d50ac8e1a0a04ddb00976f65a9151c: Status 404 returned error can't find the container with id 2e90faf8dd799960b8070f2406807a3587d50ac8e1a0a04ddb00976f65a9151c Nov 02 01:14:16.483314 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:16.483296 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 01:14:16.531877 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:16.531845 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-130.us-east-2.compute.internal\" not found" Nov 02 01:14:16.565565 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:16.565542 2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 01:14:16.632950 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:16.632920 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-130.us-east-2.compute.internal\" not found" Nov 02 01:14:16.733326 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:16.733298 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-130.us-east-2.compute.internal\" not found" Nov 02 01:14:16.772635 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:16.772607 2578 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 01:14:16.825997 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:16.825958 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-130.us-east-2.compute.internal" Nov 02 01:14:16.844235 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:16.844213 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 01:14:16.845567 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:16.845553 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-130.us-east-2.compute.internal" Nov 02 01:14:16.854966 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:16.854942 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 01:14:17.195482 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.195443 2578 apiserver.go:52] "Watching apiserver" Nov 02 01:14:17.204645 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.204601 2578 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 01:14:17.205981 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.205944 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-vcsls","kube-system/kube-apiserver-proxy-ip-10-0-136-130.us-east-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-pqmz2","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-130.us-east-2.compute.internal","openshift-ovn-kubernetes/ovnkube-node-z8bzg","openshift-network-diagnostics/network-check-target-fdptg","openshift-multus/multus-27jtx","openshift-multus/multus-additional-cni-plugins-snk78","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj5fn","openshift-dns/node-resolver-qwzn4","openshift-image-registry/node-ca-bgczm","openshift-multus/network-metrics-daemon-nkqjm","openshift-network-operator/iptables-alerter-phtn9","kube-system/global-pull-secret-syncer-kl8v7"] Nov 02 01:14:17.210116 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.209790 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qwzn4" Nov 02 01:14:17.211799 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.211777 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-phtn9" Nov 02 01:14:17.213924 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.213901 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:17.214669 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.214639 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 01:14:17.214785 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.214772 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-85fmt\"" Nov 02 01:14:17.214848 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.214825 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 01:14:17.215023 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.215006 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 01:14:17.216052 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.216011 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-67z2p\"" Nov 02 01:14:17.216150 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.216052 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 01:14:17.216150 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.216067 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 01:14:17.216493 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.216459 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kl8v7" Nov 02 01:14:17.216588 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:17.216552 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kl8v7" podUID="9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074" Nov 02 01:14:17.217465 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.217443 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 01:14:17.217725 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.217706 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 01:14:17.217869 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.217852 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vlhpz\"" Nov 02 01:14:17.219594 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.219572 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:17.222603 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.222277 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fdptg" Nov 02 01:14:17.222603 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:17.222343 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fdptg" podUID="3917d88c-88f5-4c5c-b1f7-e0195a61f88f" Nov 02 01:14:17.224753 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.224731 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj5fn" Nov 02 01:14:17.227290 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.226287 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 02 01:14:17.227518 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.227496 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-27jtx" Nov 02 01:14:17.229867 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.229846 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-snk78" Nov 02 01:14:17.232853 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.232836 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vcsls" Nov 02 01:14:17.235741 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.235719 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bgczm" Nov 02 01:14:17.237913 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.237891 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s78hk\" (UniqueName: \"kubernetes.io/projected/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-kube-api-access-s78hk\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:17.238019 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.237919 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/20bc48c0-b0ef-40df-a5c9-06c48c75fe20-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hj5fn\" (UID: \"20bc48c0-b0ef-40df-a5c9-06c48c75fe20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj5fn" Nov 02 01:14:17.238019 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.237935 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-cnibin\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:17.238019 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.237951 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-lib-modules\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:17.238019 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.237966 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/20bc48c0-b0ef-40df-a5c9-06c48c75fe20-socket-dir\") pod \"aws-ebs-csi-driver-node-hj5fn\" (UID: \"20bc48c0-b0ef-40df-a5c9-06c48c75fe20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj5fn" Nov 02 01:14:17.238019 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.237983 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-ovnkube-config\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:17.238019 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.237999 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd76d\" (UniqueName: \"kubernetes.io/projected/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-kube-api-access-pd76d\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:17.238227 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.238022 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-etc-sysctl-conf\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:17.238227 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.238059 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4a12c42f-81b2-4fde-bc01-8abe3e640957-cni-binary-copy\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:17.238227 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.238074 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-multus-socket-dir-parent\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:17.238227 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.238093 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4a12c42f-81b2-4fde-bc01-8abe3e640957-multus-daemon-config\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:17.238227 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.238116 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-tmp\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:17.238227 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.238140 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxrsj\" (UniqueName: \"kubernetes.io/projected/3917d88c-88f5-4c5c-b1f7-e0195a61f88f-kube-api-access-hxrsj\") pod \"network-check-target-fdptg\" (UID: \"3917d88c-88f5-4c5c-b1f7-e0195a61f88f\") " pod="openshift-network-diagnostics/network-check-target-fdptg" Nov 02 01:14:17.238227 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.238160 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/497c02e6-eb5a-49ed-a911-97384fb409cf-iptables-alerter-script\") pod \"iptables-alerter-phtn9\" (UID: \"497c02e6-eb5a-49ed-a911-97384fb409cf\") " pod="openshift-network-operator/iptables-alerter-phtn9" Nov 02 01:14:17.238227 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.238187 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p26j\" (UniqueName: \"kubernetes.io/projected/497c02e6-eb5a-49ed-a911-97384fb409cf-kube-api-access-4p26j\") pod \"iptables-alerter-phtn9\" (UID: \"497c02e6-eb5a-49ed-a911-97384fb409cf\") " pod="openshift-network-operator/iptables-alerter-phtn9" Nov 02 01:14:17.238227 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.238202 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-run-systemd\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:17.238227 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.238216 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-log-socket\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:17.238723 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.238232 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-env-overrides\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:17.238723 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.238264 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/20bc48c0-b0ef-40df-a5c9-06c48c75fe20-sys-fs\") pod \"aws-ebs-csi-driver-node-hj5fn\" (UID: \"20bc48c0-b0ef-40df-a5c9-06c48c75fe20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj5fn" Nov 02 01:14:17.238723 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.238310 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-ovn-node-metrics-cert\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:17.238723 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.238426 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/497c02e6-eb5a-49ed-a911-97384fb409cf-host-slash\") pod \"iptables-alerter-phtn9\" (UID: \"497c02e6-eb5a-49ed-a911-97384fb409cf\") " pod="openshift-network-operator/iptables-alerter-phtn9" Nov 02 01:14:17.238723 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.238490 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-systemd-units\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:17.238723 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.238524 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-run-ovn\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:17.238723 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.238563 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-node-log\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:17.238723 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.238608 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-etc-modprobe-d\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:17.238723 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.238654 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-host-run-k8s-cni-cncf-io\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:17.238723 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.238721 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-host-run-multus-certs\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:17.239249 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.238764 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-system-cni-dir\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:17.239249 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.238792 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-hostroot\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:17.239249 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.238840 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs7p2\" (UniqueName: \"kubernetes.io/projected/4a12c42f-81b2-4fde-bc01-8abe3e640957-kube-api-access-bs7p2\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:17.239249 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.238867 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-host-cni-netd\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:17.239249 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.238893 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-ovnkube-script-lib\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:17.239249 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.238977 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-etc-tuned\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:17.239249 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.239016 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/20bc48c0-b0ef-40df-a5c9-06c48c75fe20-registration-dir\") pod \"aws-ebs-csi-driver-node-hj5fn\" (UID: \"20bc48c0-b0ef-40df-a5c9-06c48c75fe20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj5fn" Nov 02 01:14:17.239249 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.239049 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/20bc48c0-b0ef-40df-a5c9-06c48c75fe20-device-dir\") pod \"aws-ebs-csi-driver-node-hj5fn\" (UID: \"20bc48c0-b0ef-40df-a5c9-06c48c75fe20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj5fn" Nov 02 01:14:17.239249 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.239095 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-etc-openvswitch\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:17.239249 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.239143 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-host-run-ovn-kubernetes\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:17.239249 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.239173 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-host-cni-bin\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:17.239249 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.239199 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-etc-sysconfig\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:17.239826 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.239262 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/20bc48c0-b0ef-40df-a5c9-06c48c75fe20-etc-selinux\") pod \"aws-ebs-csi-driver-node-hj5fn\" (UID: \"20bc48c0-b0ef-40df-a5c9-06c48c75fe20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj5fn" Nov 02 01:14:17.239826 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.239289 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:17.239826 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.239315 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074-dbus\") pod \"global-pull-secret-syncer-kl8v7\" (UID: \"9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074\") " pod="kube-system/global-pull-secret-syncer-kl8v7" Nov 02 01:14:17.239826 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.239344 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-host-slash\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:17.239826 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.239369 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-etc-kubernetes\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:17.239826 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.239421 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-etc-sysctl-d\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:17.239826 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.239476 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-host-var-lib-cni-multus\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:17.239826 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.239520 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-multus-conf-dir\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:17.239826 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.239551 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074-kubelet-config\") pod \"global-pull-secret-syncer-kl8v7\" (UID: \"9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074\") " pod="kube-system/global-pull-secret-syncer-kl8v7" Nov 02 01:14:17.239826 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.239589 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074-original-pull-secret\") pod \"global-pull-secret-syncer-kl8v7\" (UID: \"9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074\") " pod="kube-system/global-pull-secret-syncer-kl8v7" Nov 02 01:14:17.239826 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.239616 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-host-kubelet\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:17.239826 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.239664 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-var-lib-openvswitch\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:17.239826 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.239690 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-run\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:17.239826 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.239714 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-sys\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:17.239826 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.239739 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-host-var-lib-cni-bin\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:17.239826 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.239774 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-host-var-lib-kubelet\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:17.240328 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.239803 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/77df90af-e163-47ea-83ad-99bc1258f3d5-hosts-file\") pod \"node-resolver-qwzn4\" (UID: \"77df90af-e163-47ea-83ad-99bc1258f3d5\") " pod="openshift-dns/node-resolver-qwzn4" Nov 02 01:14:17.240328 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.239828 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-host-run-netns\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:17.240328 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.239899 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-var-lib-kubelet\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:17.240328 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.239924 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-multus-cni-dir\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:17.240328 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.239954 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-etc-kubernetes\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:17.240328 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.239996 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77df90af-e163-47ea-83ad-99bc1258f3d5-tmp-dir\") pod \"node-resolver-qwzn4\" (UID: \"77df90af-e163-47ea-83ad-99bc1258f3d5\") " pod="openshift-dns/node-resolver-qwzn4" Nov 02 01:14:17.240328 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.240024 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8tts\" (UniqueName: \"kubernetes.io/projected/77df90af-e163-47ea-83ad-99bc1258f3d5-kube-api-access-c8tts\") pod \"node-resolver-qwzn4\" (UID: \"77df90af-e163-47ea-83ad-99bc1258f3d5\") " pod="openshift-dns/node-resolver-qwzn4" Nov 02 01:14:17.240328 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.240055 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-etc-systemd\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:17.240328 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.240082 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-host-run-netns\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:17.240328 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.239999 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 01:14:17.240328 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.240143 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-run-openvswitch\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:17.240328 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.240168 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-host\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:17.240328 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.240195 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jtbq\" (UniqueName: \"kubernetes.io/projected/20bc48c0-b0ef-40df-a5c9-06c48c75fe20-kube-api-access-9jtbq\") pod \"aws-ebs-csi-driver-node-hj5fn\" (UID: \"20bc48c0-b0ef-40df-a5c9-06c48c75fe20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj5fn" Nov 02 01:14:17.240994 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.240976 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkqjm" Nov 02 01:14:17.241066 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:17.241044 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkqjm" podUID="4c7b5fa9-d0b9-4147-bc52-bdd444a2c081" Nov 02 01:14:17.264408 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.264352 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-hkbvp" Nov 02 01:14:17.274845 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.274819 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-hkbvp" Nov 02 01:14:17.323738 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.323689 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 01:14:17.323920 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.323779 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 01:14:17.323920 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.323836 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 01:14:17.323920 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.323889 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 01:14:17.325165 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.325135 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 01:14:17.325288 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.325225 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 01:14:17.325544 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.325523 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 01:14:17.325544 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.325536 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 01:14:17.325778 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.325681 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-87cgr\"" Nov 02 01:14:17.325778 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.325718 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 01:14:17.325778 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.325749 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 01:14:17.325778 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.325780 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 01:14:17.326028 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.325807 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 01:14:17.326028 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.325828 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5kn9m\"" Nov 02 01:14:17.326028 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.325844 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 01:14:17.326028 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.325864 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 01:14:17.326028 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.325903 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 01:14:17.326028 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.325906 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8rk5x\"" Nov 02 01:14:17.326028 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.325952 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-tmdq2\"" Nov 02 01:14:17.326028 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.325986 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 01:14:17.326028 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.325998 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 01:14:17.326501 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.326138 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-75lmf\"" Nov 02 01:14:17.326501 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.326181 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 01:14:17.326501 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.326204 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 01:14:17.326701 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.326678 2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 01:14:17.327546 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.327522 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7pl9l\"" Nov 02 01:14:17.327657 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.327636 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 01:14:17.340694 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.340663 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-host-run-ovn-kubernetes\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:17.340876 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.340703 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-host-cni-bin\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:17.340876 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.340734 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-etc-sysconfig\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:17.340876 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.340760 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/20bc48c0-b0ef-40df-a5c9-06c48c75fe20-etc-selinux\") pod \"aws-ebs-csi-driver-node-hj5fn\" (UID: \"20bc48c0-b0ef-40df-a5c9-06c48c75fe20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj5fn" Nov 02 01:14:17.340876 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.340795 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6a02556d-4538-478d-84cf-a4d104565c84-cni-binary-copy\") pod \"multus-additional-cni-plugins-snk78\" (UID: \"6a02556d-4538-478d-84cf-a4d104565c84\") " pod="openshift-multus/multus-additional-cni-plugins-snk78" Nov 02 01:14:17.340876 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.340809 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-host-cni-bin\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:17.340876 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.340822 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:17.340876 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.340810 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-host-run-ovn-kubernetes\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:17.340876 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.340850 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074-dbus\") pod \"global-pull-secret-syncer-kl8v7\" (UID: \"9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074\") " pod="kube-system/global-pull-secret-syncer-kl8v7" Nov 02 01:14:17.340876 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.340873 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-etc-sysconfig\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:17.341309 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.340922 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-host-slash\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:17.341309 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.340926 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:17.341309 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.340954 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/20bc48c0-b0ef-40df-a5c9-06c48c75fe20-etc-selinux\") pod \"aws-ebs-csi-driver-node-hj5fn\" (UID: \"20bc48c0-b0ef-40df-a5c9-06c48c75fe20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj5fn" Nov 02 01:14:17.341309 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.340961 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-etc-kubernetes\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:17.341309 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.340971 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-host-slash\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:17.341309 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.340997 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-etc-sysctl-d\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:17.341309 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.341015 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-host-var-lib-cni-multus\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:17.341309 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.341040 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-multus-conf-dir\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:17.341309 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.341058 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074-kubelet-config\") pod \"global-pull-secret-syncer-kl8v7\" (UID: \"9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074\") " pod="kube-system/global-pull-secret-syncer-kl8v7" Nov 02 01:14:17.341309 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.341032 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-etc-kubernetes\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:17.341309 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.341077 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-host-var-lib-cni-multus\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:17.341309 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.341123 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074-original-pull-secret\") pod \"global-pull-secret-syncer-kl8v7\" (UID: \"9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074\") " pod="kube-system/global-pull-secret-syncer-kl8v7" Nov 02 01:14:17.341309 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.341149 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-host-kubelet\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:17.341309 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.341174 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-var-lib-openvswitch\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:17.341309 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.341190 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074-dbus\") pod \"global-pull-secret-syncer-kl8v7\" (UID: \"9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074\") " pod="kube-system/global-pull-secret-syncer-kl8v7" Nov 02 01:14:17.341309 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.341213 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-multus-conf-dir\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:17.341309 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.341289 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-host-kubelet\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:17.342139 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.341357 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-run\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:17.342139 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.341404 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-sys\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:17.342139 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.341375 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-etc-sysctl-d\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:17.342139 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:17.341406 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 01:14:17.342139 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.341448 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074-kubelet-config\") pod \"global-pull-secret-syncer-kl8v7\" (UID: \"9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074\") " pod="kube-system/global-pull-secret-syncer-kl8v7" Nov 02 01:14:17.342139 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.341457 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-run\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:17.342139 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:17.341498 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074-original-pull-secret podName:9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:17.841474893 +0000 UTC m=+3.047767534 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074-original-pull-secret") pod "global-pull-secret-syncer-kl8v7" (UID: "9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074") : object "kube-system"/"original-pull-secret" not registered Nov 02 01:14:17.342139 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.341507 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-var-lib-openvswitch\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:17.342139 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.341518 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-sys\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:17.342139 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.341565 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-host-var-lib-cni-bin\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:17.342139 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.341602 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-host-var-lib-kubelet\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:17.342139 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.341645 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-host-var-lib-cni-bin\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:17.342139 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.341676 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/77df90af-e163-47ea-83ad-99bc1258f3d5-hosts-file\") pod \"node-resolver-qwzn4\" (UID: \"77df90af-e163-47ea-83ad-99bc1258f3d5\") " pod="openshift-dns/node-resolver-qwzn4" Nov 02 01:14:17.342139 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.341722 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-host-var-lib-kubelet\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:17.342139 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.341775 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-host-run-netns\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:17.342139 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.341781 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/77df90af-e163-47ea-83ad-99bc1258f3d5-hosts-file\") pod \"node-resolver-qwzn4\" (UID: \"77df90af-e163-47ea-83ad-99bc1258f3d5\") " pod="openshift-dns/node-resolver-qwzn4" Nov 02 01:14:17.342139 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.341819 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-var-lib-kubelet\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:17.342139 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.341857 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-var-lib-kubelet\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:17.343045 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.341859 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-multus-cni-dir\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:17.343045 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.341890 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-etc-kubernetes\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:17.343045 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.341927 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-host-run-netns\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:17.343045 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.341948 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-multus-cni-dir\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:17.343045 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.341971 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-etc-kubernetes\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:17.343045 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.341978 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c7b5fa9-d0b9-4147-bc52-bdd444a2c081-metrics-certs\") pod \"network-metrics-daemon-nkqjm\" (UID: \"4c7b5fa9-d0b9-4147-bc52-bdd444a2c081\") " pod="openshift-multus/network-metrics-daemon-nkqjm" Nov 02 01:14:17.343045 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.342019 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77df90af-e163-47ea-83ad-99bc1258f3d5-tmp-dir\") pod \"node-resolver-qwzn4\" (UID: \"77df90af-e163-47ea-83ad-99bc1258f3d5\") " pod="openshift-dns/node-resolver-qwzn4" Nov 02 01:14:17.343045 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.342071 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8tts\" (UniqueName: \"kubernetes.io/projected/77df90af-e163-47ea-83ad-99bc1258f3d5-kube-api-access-c8tts\") pod \"node-resolver-qwzn4\" (UID: \"77df90af-e163-47ea-83ad-99bc1258f3d5\") " pod="openshift-dns/node-resolver-qwzn4" Nov 02 01:14:17.343045 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.342095 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-etc-systemd\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:17.343045 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.342119 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-host-run-netns\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:17.343045 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.342148 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6a02556d-4538-478d-84cf-a4d104565c84-cnibin\") pod \"multus-additional-cni-plugins-snk78\" (UID: \"6a02556d-4538-478d-84cf-a4d104565c84\") " pod="openshift-multus/multus-additional-cni-plugins-snk78" Nov 02 01:14:17.343045 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.342161 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-etc-systemd\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:17.343045 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.342173 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b98jt\" (UniqueName: \"kubernetes.io/projected/4c7b5fa9-d0b9-4147-bc52-bdd444a2c081-kube-api-access-b98jt\") pod \"network-metrics-daemon-nkqjm\" (UID: \"4c7b5fa9-d0b9-4147-bc52-bdd444a2c081\") " pod="openshift-multus/network-metrics-daemon-nkqjm" Nov 02 01:14:17.343045 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.342199 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-run-openvswitch\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:17.343045 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.342223 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-host\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:17.343045 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.342240 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-host-run-netns\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:17.343045 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.342250 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jtbq\" (UniqueName: \"kubernetes.io/projected/20bc48c0-b0ef-40df-a5c9-06c48c75fe20-kube-api-access-9jtbq\") pod \"aws-ebs-csi-driver-node-hj5fn\" (UID: \"20bc48c0-b0ef-40df-a5c9-06c48c75fe20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj5fn" Nov 02 01:14:17.343045 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.342315 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-run-openvswitch\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:17.343942 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.342321 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-host\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:17.343942 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.342393 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s78hk\" (UniqueName: \"kubernetes.io/projected/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-kube-api-access-s78hk\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:17.343942 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.342446 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/20bc48c0-b0ef-40df-a5c9-06c48c75fe20-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hj5fn\" (UID: \"20bc48c0-b0ef-40df-a5c9-06c48c75fe20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj5fn" Nov 02 01:14:17.343942 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.342477 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-cnibin\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:17.343942 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.342506 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/16eed615-a448-4416-b06e-1c33d1fffb6d-agent-certs\") pod \"konnectivity-agent-vcsls\" (UID: \"16eed615-a448-4416-b06e-1c33d1fffb6d\") " pod="kube-system/konnectivity-agent-vcsls" Nov 02 01:14:17.343942 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.342533 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/16eed615-a448-4416-b06e-1c33d1fffb6d-konnectivity-ca\") pod \"konnectivity-agent-vcsls\" (UID: \"16eed615-a448-4416-b06e-1c33d1fffb6d\") " pod="kube-system/konnectivity-agent-vcsls" Nov 02 01:14:17.343942 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.342555 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77df90af-e163-47ea-83ad-99bc1258f3d5-tmp-dir\") pod \"node-resolver-qwzn4\" (UID: \"77df90af-e163-47ea-83ad-99bc1258f3d5\") " pod="openshift-dns/node-resolver-qwzn4" Nov 02 01:14:17.343942 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.342576 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/41f79a83-9886-4c59-a904-328463dfe674-serviceca\") pod \"node-ca-bgczm\" (UID: \"41f79a83-9886-4c59-a904-328463dfe674\") " pod="openshift-image-registry/node-ca-bgczm" Nov 02 01:14:17.343942 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.342579 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-cnibin\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:17.343942 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.342614 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/20bc48c0-b0ef-40df-a5c9-06c48c75fe20-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hj5fn\" (UID: \"20bc48c0-b0ef-40df-a5c9-06c48c75fe20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj5fn" Nov 02 01:14:17.343942 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.342678 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-lib-modules\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:17.343942 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.342697 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/20bc48c0-b0ef-40df-a5c9-06c48c75fe20-socket-dir\") pod \"aws-ebs-csi-driver-node-hj5fn\" (UID: \"20bc48c0-b0ef-40df-a5c9-06c48c75fe20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj5fn" Nov 02 01:14:17.343942 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.342740 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a02556d-4538-478d-84cf-a4d104565c84-system-cni-dir\") pod \"multus-additional-cni-plugins-snk78\" (UID: \"6a02556d-4538-478d-84cf-a4d104565c84\") " pod="openshift-multus/multus-additional-cni-plugins-snk78" Nov 02 01:14:17.343942 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.342852 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-ovnkube-config\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:17.343942 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.342850 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/20bc48c0-b0ef-40df-a5c9-06c48c75fe20-socket-dir\") pod \"aws-ebs-csi-driver-node-hj5fn\" (UID: \"20bc48c0-b0ef-40df-a5c9-06c48c75fe20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj5fn" Nov 02 01:14:17.343942 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.342904 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd76d\" (UniqueName: \"kubernetes.io/projected/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-kube-api-access-pd76d\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:17.343942 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.342929 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-etc-sysctl-conf\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:17.344855 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.342992 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-lib-modules\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:17.344855 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.343030 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4a12c42f-81b2-4fde-bc01-8abe3e640957-cni-binary-copy\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:17.344855 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.343062 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-etc-sysctl-conf\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:17.344855 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.343105 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-multus-socket-dir-parent\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:17.344855 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.343188 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4a12c42f-81b2-4fde-bc01-8abe3e640957-multus-daemon-config\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:17.344855 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.343227 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-multus-socket-dir-parent\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:17.344855 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.343259 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-tmp\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:17.344855 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.343279 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxrsj\" (UniqueName: \"kubernetes.io/projected/3917d88c-88f5-4c5c-b1f7-e0195a61f88f-kube-api-access-hxrsj\") pod \"network-check-target-fdptg\" (UID: \"3917d88c-88f5-4c5c-b1f7-e0195a61f88f\") " pod="openshift-network-diagnostics/network-check-target-fdptg" Nov 02 01:14:17.344855 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.343322 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/497c02e6-eb5a-49ed-a911-97384fb409cf-iptables-alerter-script\") pod \"iptables-alerter-phtn9\" (UID: \"497c02e6-eb5a-49ed-a911-97384fb409cf\") " pod="openshift-network-operator/iptables-alerter-phtn9" Nov 02 01:14:17.344855 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.343352 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p26j\" (UniqueName: \"kubernetes.io/projected/497c02e6-eb5a-49ed-a911-97384fb409cf-kube-api-access-4p26j\") pod \"iptables-alerter-phtn9\" (UID: \"497c02e6-eb5a-49ed-a911-97384fb409cf\") " pod="openshift-network-operator/iptables-alerter-phtn9" Nov 02 01:14:17.344855 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.343425 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-run-systemd\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:17.344855 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.343461 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khqc7\" (UniqueName: \"kubernetes.io/projected/41f79a83-9886-4c59-a904-328463dfe674-kube-api-access-khqc7\") pod \"node-ca-bgczm\" (UID: \"41f79a83-9886-4c59-a904-328463dfe674\") " pod="openshift-image-registry/node-ca-bgczm" Nov 02 01:14:17.344855 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.343488 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-log-socket\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:17.344855 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.343592 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-env-overrides\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:17.344855 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.343619 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/20bc48c0-b0ef-40df-a5c9-06c48c75fe20-sys-fs\") pod \"aws-ebs-csi-driver-node-hj5fn\" (UID: \"20bc48c0-b0ef-40df-a5c9-06c48c75fe20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj5fn" Nov 02 01:14:17.344855 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.343630 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-run-systemd\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:17.344855 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.343646 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-ovn-node-metrics-cert\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:17.345780 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.343868 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-log-socket\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:17.345780 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.343971 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41f79a83-9886-4c59-a904-328463dfe674-host\") pod \"node-ca-bgczm\" (UID: \"41f79a83-9886-4c59-a904-328463dfe674\") " pod="openshift-image-registry/node-ca-bgczm" Nov 02 01:14:17.345780 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.343996 2578 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 01:14:17.345780 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.344011 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6a02556d-4538-478d-84cf-a4d104565c84-tuning-conf-dir\") pod \"multus-additional-cni-plugins-snk78\" (UID: \"6a02556d-4538-478d-84cf-a4d104565c84\") " pod="openshift-multus/multus-additional-cni-plugins-snk78" Nov 02 01:14:17.345780 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.344040 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6a02556d-4538-478d-84cf-a4d104565c84-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-snk78\" (UID: \"6a02556d-4538-478d-84cf-a4d104565c84\") " pod="openshift-multus/multus-additional-cni-plugins-snk78" Nov 02 01:14:17.345780 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.344102 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-ovnkube-config\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:17.345780 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.344167 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/497c02e6-eb5a-49ed-a911-97384fb409cf-host-slash\") pod \"iptables-alerter-phtn9\" (UID: \"497c02e6-eb5a-49ed-a911-97384fb409cf\") " pod="openshift-network-operator/iptables-alerter-phtn9" Nov 02 01:14:17.345780 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.344186 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-systemd-units\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:17.345780 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.344213 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-run-ovn\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:17.345780 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.344228 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-node-log\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:17.345780 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.344261 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-etc-modprobe-d\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:17.345780 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.344279 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-host-run-k8s-cni-cncf-io\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:17.345780 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.344307 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-host-run-multus-certs\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:17.345780 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.344372 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-system-cni-dir\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:17.345780 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.344400 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-hostroot\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:17.345780 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.344419 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-env-overrides\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:17.345780 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.344424 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs7p2\" (UniqueName: \"kubernetes.io/projected/4a12c42f-81b2-4fde-bc01-8abe3e640957-kube-api-access-bs7p2\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:17.345780 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.344510 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlbbj\" (UniqueName: \"kubernetes.io/projected/6a02556d-4538-478d-84cf-a4d104565c84-kube-api-access-nlbbj\") pod \"multus-additional-cni-plugins-snk78\" (UID: \"6a02556d-4538-478d-84cf-a4d104565c84\") " pod="openshift-multus/multus-additional-cni-plugins-snk78" Nov 02 01:14:17.346715 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.344545 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-host-cni-netd\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:17.346715 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.344573 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-ovnkube-script-lib\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:17.346715 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.344601 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-etc-tuned\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:17.346715 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.344619 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-host-run-multus-certs\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:17.346715 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.344661 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/20bc48c0-b0ef-40df-a5c9-06c48c75fe20-sys-fs\") pod \"aws-ebs-csi-driver-node-hj5fn\" (UID: \"20bc48c0-b0ef-40df-a5c9-06c48c75fe20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj5fn" Nov 02 01:14:17.346715 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.344685 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/497c02e6-eb5a-49ed-a911-97384fb409cf-host-slash\") pod \"iptables-alerter-phtn9\" (UID: \"497c02e6-eb5a-49ed-a911-97384fb409cf\") " pod="openshift-network-operator/iptables-alerter-phtn9" Nov 02 01:14:17.346715 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.344707 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-run-ovn\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:17.346715 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.344729 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-node-log\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:17.346715 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.344786 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-etc-modprobe-d\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:17.346715 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.344798 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4a12c42f-81b2-4fde-bc01-8abe3e640957-cni-binary-copy\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:17.346715 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.344808 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-host-run-k8s-cni-cncf-io\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:17.346715 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.344845 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-system-cni-dir\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:17.346715 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.344848 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4a12c42f-81b2-4fde-bc01-8abe3e640957-multus-daemon-config\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:17.346715 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.344894 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-hostroot\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:17.346715 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.344929 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/497c02e6-eb5a-49ed-a911-97384fb409cf-iptables-alerter-script\") pod \"iptables-alerter-phtn9\" (UID: \"497c02e6-eb5a-49ed-a911-97384fb409cf\") " pod="openshift-network-operator/iptables-alerter-phtn9" Nov 02 01:14:17.346715 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.344952 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-host-cni-netd\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:17.346715 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.344981 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/20bc48c0-b0ef-40df-a5c9-06c48c75fe20-registration-dir\") pod \"aws-ebs-csi-driver-node-hj5fn\" (UID: \"20bc48c0-b0ef-40df-a5c9-06c48c75fe20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj5fn" Nov 02 01:14:17.346715 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.345000 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/20bc48c0-b0ef-40df-a5c9-06c48c75fe20-device-dir\") pod \"aws-ebs-csi-driver-node-hj5fn\" (UID: \"20bc48c0-b0ef-40df-a5c9-06c48c75fe20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj5fn" Nov 02 01:14:17.347664 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.345011 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/20bc48c0-b0ef-40df-a5c9-06c48c75fe20-registration-dir\") pod \"aws-ebs-csi-driver-node-hj5fn\" (UID: \"20bc48c0-b0ef-40df-a5c9-06c48c75fe20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj5fn" Nov 02 01:14:17.347664 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.345018 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-systemd-units\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:17.347664 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.345031 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6a02556d-4538-478d-84cf-a4d104565c84-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-snk78\" (UID: \"6a02556d-4538-478d-84cf-a4d104565c84\") " pod="openshift-multus/multus-additional-cni-plugins-snk78" Nov 02 01:14:17.347664 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.345053 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-etc-openvswitch\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:17.347664 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.345078 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-etc-openvswitch\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:17.347664 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.345136 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/20bc48c0-b0ef-40df-a5c9-06c48c75fe20-device-dir\") pod \"aws-ebs-csi-driver-node-hj5fn\" (UID: \"20bc48c0-b0ef-40df-a5c9-06c48c75fe20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj5fn" Nov 02 01:14:17.347664 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.345418 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-ovnkube-script-lib\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:17.349393 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.349368 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-ovn-node-metrics-cert\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:17.349751 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.349711 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-etc-tuned\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:17.350885 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.350861 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-tmp\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:17.355660 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:17.354622 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 01:14:17.355660 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:17.354646 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 01:14:17.355660 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:17.354660 2578 projected.go:196] Error preparing data for projected volume kube-api-access-hxrsj for pod openshift-network-diagnostics/network-check-target-fdptg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 01:14:17.355660 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:17.354728 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3917d88c-88f5-4c5c-b1f7-e0195a61f88f-kube-api-access-hxrsj podName:3917d88c-88f5-4c5c-b1f7-e0195a61f88f nodeName:}" failed. No retries permitted until 2026-11-02 01:14:17.854709535 +0000 UTC m=+3.061002198 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hxrsj" (UniqueName: "kubernetes.io/projected/3917d88c-88f5-4c5c-b1f7-e0195a61f88f-kube-api-access-hxrsj") pod "network-check-target-fdptg" (UID: "3917d88c-88f5-4c5c-b1f7-e0195a61f88f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 01:14:17.359394 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.359369 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs7p2\" (UniqueName: \"kubernetes.io/projected/4a12c42f-81b2-4fde-bc01-8abe3e640957-kube-api-access-bs7p2\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:17.359581 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.359555 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p26j\" (UniqueName: \"kubernetes.io/projected/497c02e6-eb5a-49ed-a911-97384fb409cf-kube-api-access-4p26j\") pod \"iptables-alerter-phtn9\" (UID: \"497c02e6-eb5a-49ed-a911-97384fb409cf\") " pod="openshift-network-operator/iptables-alerter-phtn9" Nov 02 01:14:17.361119 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.361089 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd76d\" (UniqueName: \"kubernetes.io/projected/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-kube-api-access-pd76d\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:17.361210 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.361118 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jtbq\" (UniqueName: \"kubernetes.io/projected/20bc48c0-b0ef-40df-a5c9-06c48c75fe20-kube-api-access-9jtbq\") pod \"aws-ebs-csi-driver-node-hj5fn\" (UID: \"20bc48c0-b0ef-40df-a5c9-06c48c75fe20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj5fn" Nov 02 01:14:17.362046 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.361538 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s78hk\" (UniqueName: \"kubernetes.io/projected/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-kube-api-access-s78hk\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:17.362046 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.361971 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8tts\" (UniqueName: \"kubernetes.io/projected/77df90af-e163-47ea-83ad-99bc1258f3d5-kube-api-access-c8tts\") pod \"node-resolver-qwzn4\" (UID: \"77df90af-e163-47ea-83ad-99bc1258f3d5\") " pod="openshift-dns/node-resolver-qwzn4" Nov 02 01:14:17.370304 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.370237 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-130.us-east-2.compute.internal" event={"ID":"b96c82052e4f897819ee4482b034eaf4","Type":"ContainerStarted","Data":"3d2bf19f263abe4ecec40ed8a73450e0a6ece52303e4d7e99934da5b998f2b5b"} Nov 02 01:14:17.371616 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.371593 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-130.us-east-2.compute.internal" event={"ID":"dabfce80149e41ad659d129c965945bf","Type":"ContainerStarted","Data":"2e90faf8dd799960b8070f2406807a3587d50ac8e1a0a04ddb00976f65a9151c"} Nov 02 01:14:17.419503 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.419460 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 01:14:17.446253 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.446162 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6a02556d-4538-478d-84cf-a4d104565c84-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-snk78\" (UID: \"6a02556d-4538-478d-84cf-a4d104565c84\") " pod="openshift-multus/multus-additional-cni-plugins-snk78" Nov 02 01:14:17.446253 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.446222 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6a02556d-4538-478d-84cf-a4d104565c84-cni-binary-copy\") pod \"multus-additional-cni-plugins-snk78\" (UID: \"6a02556d-4538-478d-84cf-a4d104565c84\") " pod="openshift-multus/multus-additional-cni-plugins-snk78" Nov 02 01:14:17.446518 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.446279 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c7b5fa9-d0b9-4147-bc52-bdd444a2c081-metrics-certs\") pod \"network-metrics-daemon-nkqjm\" (UID: \"4c7b5fa9-d0b9-4147-bc52-bdd444a2c081\") " pod="openshift-multus/network-metrics-daemon-nkqjm" Nov 02 01:14:17.446518 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.446316 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6a02556d-4538-478d-84cf-a4d104565c84-cnibin\") pod \"multus-additional-cni-plugins-snk78\" (UID: \"6a02556d-4538-478d-84cf-a4d104565c84\") " pod="openshift-multus/multus-additional-cni-plugins-snk78" Nov 02 01:14:17.446518 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.446342 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b98jt\" (UniqueName: \"kubernetes.io/projected/4c7b5fa9-d0b9-4147-bc52-bdd444a2c081-kube-api-access-b98jt\") pod \"network-metrics-daemon-nkqjm\" (UID: \"4c7b5fa9-d0b9-4147-bc52-bdd444a2c081\") " pod="openshift-multus/network-metrics-daemon-nkqjm" Nov 02 01:14:17.446518 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.446370 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/16eed615-a448-4416-b06e-1c33d1fffb6d-agent-certs\") pod \"konnectivity-agent-vcsls\" (UID: \"16eed615-a448-4416-b06e-1c33d1fffb6d\") " pod="kube-system/konnectivity-agent-vcsls" Nov 02 01:14:17.446518 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.446410 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/16eed615-a448-4416-b06e-1c33d1fffb6d-konnectivity-ca\") pod \"konnectivity-agent-vcsls\" (UID: \"16eed615-a448-4416-b06e-1c33d1fffb6d\") " pod="kube-system/konnectivity-agent-vcsls" Nov 02 01:14:17.446518 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.446436 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/41f79a83-9886-4c59-a904-328463dfe674-serviceca\") pod \"node-ca-bgczm\" (UID: \"41f79a83-9886-4c59-a904-328463dfe674\") " pod="openshift-image-registry/node-ca-bgczm" Nov 02 01:14:17.446772 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:17.446555 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 01:14:17.446772 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:17.446621 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c7b5fa9-d0b9-4147-bc52-bdd444a2c081-metrics-certs podName:4c7b5fa9-d0b9-4147-bc52-bdd444a2c081 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:17.946602481 +0000 UTC m=+3.152895142 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4c7b5fa9-d0b9-4147-bc52-bdd444a2c081-metrics-certs") pod "network-metrics-daemon-nkqjm" (UID: "4c7b5fa9-d0b9-4147-bc52-bdd444a2c081") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 01:14:17.446772 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.446692 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a02556d-4538-478d-84cf-a4d104565c84-system-cni-dir\") pod \"multus-additional-cni-plugins-snk78\" (UID: \"6a02556d-4538-478d-84cf-a4d104565c84\") " pod="openshift-multus/multus-additional-cni-plugins-snk78" Nov 02 01:14:17.446772 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.446767 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khqc7\" (UniqueName: \"kubernetes.io/projected/41f79a83-9886-4c59-a904-328463dfe674-kube-api-access-khqc7\") pod \"node-ca-bgczm\" (UID: \"41f79a83-9886-4c59-a904-328463dfe674\") " pod="openshift-image-registry/node-ca-bgczm" Nov 02 01:14:17.447005 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.446798 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41f79a83-9886-4c59-a904-328463dfe674-host\") pod \"node-ca-bgczm\" (UID: \"41f79a83-9886-4c59-a904-328463dfe674\") " pod="openshift-image-registry/node-ca-bgczm" Nov 02 01:14:17.447005 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.446825 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6a02556d-4538-478d-84cf-a4d104565c84-cnibin\") pod \"multus-additional-cni-plugins-snk78\" (UID: \"6a02556d-4538-478d-84cf-a4d104565c84\") " pod="openshift-multus/multus-additional-cni-plugins-snk78" Nov 02 01:14:17.447005 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.446827 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6a02556d-4538-478d-84cf-a4d104565c84-tuning-conf-dir\") pod \"multus-additional-cni-plugins-snk78\" (UID: \"6a02556d-4538-478d-84cf-a4d104565c84\") " pod="openshift-multus/multus-additional-cni-plugins-snk78" Nov 02 01:14:17.447005 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.446866 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6a02556d-4538-478d-84cf-a4d104565c84-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-snk78\" (UID: \"6a02556d-4538-478d-84cf-a4d104565c84\") " pod="openshift-multus/multus-additional-cni-plugins-snk78" Nov 02 01:14:17.447005 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.446908 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlbbj\" (UniqueName: \"kubernetes.io/projected/6a02556d-4538-478d-84cf-a4d104565c84-kube-api-access-nlbbj\") pod \"multus-additional-cni-plugins-snk78\" (UID: \"6a02556d-4538-478d-84cf-a4d104565c84\") " pod="openshift-multus/multus-additional-cni-plugins-snk78" Nov 02 01:14:17.447523 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.447392 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a02556d-4538-478d-84cf-a4d104565c84-system-cni-dir\") pod \"multus-additional-cni-plugins-snk78\" (UID: \"6a02556d-4538-478d-84cf-a4d104565c84\") " pod="openshift-multus/multus-additional-cni-plugins-snk78" Nov 02 01:14:17.447523 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.447456 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/41f79a83-9886-4c59-a904-328463dfe674-serviceca\") pod \"node-ca-bgczm\" (UID: \"41f79a83-9886-4c59-a904-328463dfe674\") " pod="openshift-image-registry/node-ca-bgczm" Nov 02 01:14:17.447523 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.447521 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41f79a83-9886-4c59-a904-328463dfe674-host\") pod \"node-ca-bgczm\" (UID: \"41f79a83-9886-4c59-a904-328463dfe674\") " pod="openshift-image-registry/node-ca-bgczm" Nov 02 01:14:17.447754 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.447550 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6a02556d-4538-478d-84cf-a4d104565c84-cni-binary-copy\") pod \"multus-additional-cni-plugins-snk78\" (UID: \"6a02556d-4538-478d-84cf-a4d104565c84\") " pod="openshift-multus/multus-additional-cni-plugins-snk78" Nov 02 01:14:17.447754 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.447559 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6a02556d-4538-478d-84cf-a4d104565c84-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-snk78\" (UID: \"6a02556d-4538-478d-84cf-a4d104565c84\") " pod="openshift-multus/multus-additional-cni-plugins-snk78" Nov 02 01:14:17.447754 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.447726 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6a02556d-4538-478d-84cf-a4d104565c84-tuning-conf-dir\") pod \"multus-additional-cni-plugins-snk78\" (UID: \"6a02556d-4538-478d-84cf-a4d104565c84\") " pod="openshift-multus/multus-additional-cni-plugins-snk78" Nov 02 01:14:17.447899 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.447795 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6a02556d-4538-478d-84cf-a4d104565c84-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-snk78\" (UID: \"6a02556d-4538-478d-84cf-a4d104565c84\") " pod="openshift-multus/multus-additional-cni-plugins-snk78" Nov 02 01:14:17.448730 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.448682 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/16eed615-a448-4416-b06e-1c33d1fffb6d-konnectivity-ca\") pod \"konnectivity-agent-vcsls\" (UID: \"16eed615-a448-4416-b06e-1c33d1fffb6d\") " pod="kube-system/konnectivity-agent-vcsls" Nov 02 01:14:17.450560 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.450540 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/16eed615-a448-4416-b06e-1c33d1fffb6d-agent-certs\") pod \"konnectivity-agent-vcsls\" (UID: \"16eed615-a448-4416-b06e-1c33d1fffb6d\") " pod="kube-system/konnectivity-agent-vcsls" Nov 02 01:14:17.460716 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.460688 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlbbj\" (UniqueName: \"kubernetes.io/projected/6a02556d-4538-478d-84cf-a4d104565c84-kube-api-access-nlbbj\") pod \"multus-additional-cni-plugins-snk78\" (UID: \"6a02556d-4538-478d-84cf-a4d104565c84\") " pod="openshift-multus/multus-additional-cni-plugins-snk78" Nov 02 01:14:17.461179 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.461154 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b98jt\" (UniqueName: \"kubernetes.io/projected/4c7b5fa9-d0b9-4147-bc52-bdd444a2c081-kube-api-access-b98jt\") pod \"network-metrics-daemon-nkqjm\" (UID: \"4c7b5fa9-d0b9-4147-bc52-bdd444a2c081\") " pod="openshift-multus/network-metrics-daemon-nkqjm" Nov 02 01:14:17.461536 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.461490 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khqc7\" (UniqueName: \"kubernetes.io/projected/41f79a83-9886-4c59-a904-328463dfe674-kube-api-access-khqc7\") pod \"node-ca-bgczm\" (UID: \"41f79a83-9886-4c59-a904-328463dfe674\") " pod="openshift-image-registry/node-ca-bgczm" Nov 02 01:14:17.528889 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.528788 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qwzn4" Nov 02 01:14:17.538502 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.538473 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-phtn9" Nov 02 01:14:17.539027 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:17.538995 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77df90af_e163_47ea_83ad_99bc1258f3d5.slice/crio-6bfb4f88c1ee195af16386275dfe6d1435fbf25e100db3542fc0ea3491d09763 WatchSource:0}: Error finding container 6bfb4f88c1ee195af16386275dfe6d1435fbf25e100db3542fc0ea3491d09763: Status 404 returned error can't find the container with id 6bfb4f88c1ee195af16386275dfe6d1435fbf25e100db3542fc0ea3491d09763 Nov 02 01:14:17.549188 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:17.549159 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod497c02e6_eb5a_49ed_a911_97384fb409cf.slice/crio-ae3076e51de4d109bea4f96e56ae9078cdc1e253651ff937b6412f2a75fdca73 WatchSource:0}: Error finding container ae3076e51de4d109bea4f96e56ae9078cdc1e253651ff937b6412f2a75fdca73: Status 404 returned error can't find the container with id ae3076e51de4d109bea4f96e56ae9078cdc1e253651ff937b6412f2a75fdca73 Nov 02 01:14:17.550053 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.550030 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:17.560421 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:17.560392 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cd0f6e2_e491_4b6c_943a_c1115300dcaf.slice/crio-42c3f5cf41d30d21fa23a6b14283f26558cf609222d9b8b9220f52aaac2dcf27 WatchSource:0}: Error finding container 42c3f5cf41d30d21fa23a6b14283f26558cf609222d9b8b9220f52aaac2dcf27: Status 404 returned error can't find the container with id 42c3f5cf41d30d21fa23a6b14283f26558cf609222d9b8b9220f52aaac2dcf27 Nov 02 01:14:17.580788 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.580732 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:17.584796 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.584768 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-27jtx" Nov 02 01:14:17.587305 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.587220 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj5fn" Nov 02 01:14:17.593336 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:17.593296 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84e88ab0_7616_4917_ad8f_ae3c8b5a0ee7.slice/crio-b57aace4925349b187d15ca591ceaa19f18b71dc50ad47e77946b14bfe8eced1 WatchSource:0}: Error finding container b57aace4925349b187d15ca591ceaa19f18b71dc50ad47e77946b14bfe8eced1: Status 404 returned error can't find the container with id b57aace4925349b187d15ca591ceaa19f18b71dc50ad47e77946b14bfe8eced1 Nov 02 01:14:17.597556 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:17.597529 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a12c42f_81b2_4fde_bc01_8abe3e640957.slice/crio-b83ec72b85125f1dd4815579f8c600c42c52c752248fa07b679bdf6f9e33efcc WatchSource:0}: Error finding container b83ec72b85125f1dd4815579f8c600c42c52c752248fa07b679bdf6f9e33efcc: Status 404 returned error can't find the container with id b83ec72b85125f1dd4815579f8c600c42c52c752248fa07b679bdf6f9e33efcc Nov 02 01:14:17.601255 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.601235 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vcsls" Nov 02 01:14:17.601872 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:17.601842 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20bc48c0_b0ef_40df_a5c9_06c48c75fe20.slice/crio-2b3edccbd7f2f3eb5e26274f63117faabff182f22a6494b0d33aa75cbb412744 WatchSource:0}: Error finding container 2b3edccbd7f2f3eb5e26274f63117faabff182f22a6494b0d33aa75cbb412744: Status 404 returned error can't find the container with id 2b3edccbd7f2f3eb5e26274f63117faabff182f22a6494b0d33aa75cbb412744 Nov 02 01:14:17.602951 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.602929 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bgczm" Nov 02 01:14:17.610546 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.610522 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-snk78" Nov 02 01:14:17.615755 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:17.615704 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16eed615_a448_4416_b06e_1c33d1fffb6d.slice/crio-5ff16da0d1adf82e4e308b04c68653d0bb4053329ddfd88d31a058ef055e5d46 WatchSource:0}: Error finding container 5ff16da0d1adf82e4e308b04c68653d0bb4053329ddfd88d31a058ef055e5d46: Status 404 returned error can't find the container with id 5ff16da0d1adf82e4e308b04c68653d0bb4053329ddfd88d31a058ef055e5d46 Nov 02 01:14:17.619428 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:17.619361 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f79a83_9886_4c59_a904_328463dfe674.slice/crio-f4a612f867153473c11bcddf7599b2f0e9e4df5ca9edf4801dbf9bd8e5090aa2 WatchSource:0}: Error finding container f4a612f867153473c11bcddf7599b2f0e9e4df5ca9edf4801dbf9bd8e5090aa2: Status 404 returned error can't find the container with id f4a612f867153473c11bcddf7599b2f0e9e4df5ca9edf4801dbf9bd8e5090aa2 Nov 02 01:14:17.622698 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:17.622668 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a02556d_4538_478d_84cf_a4d104565c84.slice/crio-edab9d76ebb11f8f2d831955285975137ed516cc47d9a4b26d8812adad4ea7e3 WatchSource:0}: Error finding container edab9d76ebb11f8f2d831955285975137ed516cc47d9a4b26d8812adad4ea7e3: Status 404 returned error can't find the container with id edab9d76ebb11f8f2d831955285975137ed516cc47d9a4b26d8812adad4ea7e3 Nov 02 01:14:17.851108 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.851007 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074-original-pull-secret\") pod \"global-pull-secret-syncer-kl8v7\" (UID: \"9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074\") " pod="kube-system/global-pull-secret-syncer-kl8v7" Nov 02 01:14:17.851288 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:17.851188 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 01:14:17.851288 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:17.851239 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074-original-pull-secret podName:9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:18.851226622 +0000 UTC m=+4.057519278 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074-original-pull-secret") pod "global-pull-secret-syncer-kl8v7" (UID: "9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074") : object "kube-system"/"original-pull-secret" not registered Nov 02 01:14:17.951452 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.951390 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c7b5fa9-d0b9-4147-bc52-bdd444a2c081-metrics-certs\") pod \"network-metrics-daemon-nkqjm\" (UID: \"4c7b5fa9-d0b9-4147-bc52-bdd444a2c081\") " pod="openshift-multus/network-metrics-daemon-nkqjm" Nov 02 01:14:17.951642 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:17.951484 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxrsj\" (UniqueName: \"kubernetes.io/projected/3917d88c-88f5-4c5c-b1f7-e0195a61f88f-kube-api-access-hxrsj\") pod \"network-check-target-fdptg\" (UID: \"3917d88c-88f5-4c5c-b1f7-e0195a61f88f\") " pod="openshift-network-diagnostics/network-check-target-fdptg" Nov 02 01:14:17.951642 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:17.951565 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 01:14:17.951642 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:17.951630 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c7b5fa9-d0b9-4147-bc52-bdd444a2c081-metrics-certs podName:4c7b5fa9-d0b9-4147-bc52-bdd444a2c081 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:18.951616227 +0000 UTC m=+4.157908903 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4c7b5fa9-d0b9-4147-bc52-bdd444a2c081-metrics-certs") pod "network-metrics-daemon-nkqjm" (UID: "4c7b5fa9-d0b9-4147-bc52-bdd444a2c081") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 01:14:17.951642 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:17.951634 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 01:14:17.951861 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:17.951654 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 01:14:17.951861 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:17.951667 2578 projected.go:196] Error preparing data for projected volume kube-api-access-hxrsj for pod openshift-network-diagnostics/network-check-target-fdptg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 01:14:17.951861 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:17.951718 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3917d88c-88f5-4c5c-b1f7-e0195a61f88f-kube-api-access-hxrsj podName:3917d88c-88f5-4c5c-b1f7-e0195a61f88f nodeName:}" failed. No retries permitted until 2026-11-02 01:14:18.951700764 +0000 UTC m=+4.157993411 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hxrsj" (UniqueName: "kubernetes.io/projected/3917d88c-88f5-4c5c-b1f7-e0195a61f88f-kube-api-access-hxrsj") pod "network-check-target-fdptg" (UID: "3917d88c-88f5-4c5c-b1f7-e0195a61f88f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 01:14:18.278853 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:18.278648 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 01:09:17 +0000 UTC" deadline="2028-08-15 08:57:36.086525074 +0000 UTC" Nov 02 01:14:18.278853 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:18.278710 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15655h43m17.807821491s" Nov 02 01:14:18.366217 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:18.365625 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fdptg" Nov 02 01:14:18.366217 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:18.365760 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fdptg" podUID="3917d88c-88f5-4c5c-b1f7-e0195a61f88f" Nov 02 01:14:18.376251 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:18.376151 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-27jtx" event={"ID":"4a12c42f-81b2-4fde-bc01-8abe3e640957","Type":"ContainerStarted","Data":"b83ec72b85125f1dd4815579f8c600c42c52c752248fa07b679bdf6f9e33efcc"} Nov 02 01:14:18.384449 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:18.384411 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" event={"ID":"8cd0f6e2-e491-4b6c-943a-c1115300dcaf","Type":"ContainerStarted","Data":"42c3f5cf41d30d21fa23a6b14283f26558cf609222d9b8b9220f52aaac2dcf27"} Nov 02 01:14:18.390560 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:18.390524 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-phtn9" event={"ID":"497c02e6-eb5a-49ed-a911-97384fb409cf","Type":"ContainerStarted","Data":"ae3076e51de4d109bea4f96e56ae9078cdc1e253651ff937b6412f2a75fdca73"} Nov 02 01:14:18.393837 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:18.393806 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qwzn4" event={"ID":"77df90af-e163-47ea-83ad-99bc1258f3d5","Type":"ContainerStarted","Data":"6bfb4f88c1ee195af16386275dfe6d1435fbf25e100db3542fc0ea3491d09763"} Nov 02 01:14:18.399498 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:18.398932 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-130.us-east-2.compute.internal" event={"ID":"dabfce80149e41ad659d129c965945bf","Type":"ContainerStarted","Data":"4bdf24dc6473df78ee8e7e673408f85f99f49c91250776dd1af836e084343c4f"} Nov 02 01:14:18.401571 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:18.401504 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-snk78" event={"ID":"6a02556d-4538-478d-84cf-a4d104565c84","Type":"ContainerStarted","Data":"edab9d76ebb11f8f2d831955285975137ed516cc47d9a4b26d8812adad4ea7e3"} Nov 02 01:14:18.404423 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:18.404333 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bgczm" event={"ID":"41f79a83-9886-4c59-a904-328463dfe674","Type":"ContainerStarted","Data":"f4a612f867153473c11bcddf7599b2f0e9e4df5ca9edf4801dbf9bd8e5090aa2"} Nov 02 01:14:18.414342 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:18.414272 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" event={"ID":"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7","Type":"ContainerStarted","Data":"b57aace4925349b187d15ca591ceaa19f18b71dc50ad47e77946b14bfe8eced1"} Nov 02 01:14:18.417208 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:18.417177 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vcsls" event={"ID":"16eed615-a448-4416-b06e-1c33d1fffb6d","Type":"ContainerStarted","Data":"5ff16da0d1adf82e4e308b04c68653d0bb4053329ddfd88d31a058ef055e5d46"} Nov 02 01:14:18.425198 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:18.425167 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj5fn" event={"ID":"20bc48c0-b0ef-40df-a5c9-06c48c75fe20","Type":"ContainerStarted","Data":"2b3edccbd7f2f3eb5e26274f63117faabff182f22a6494b0d33aa75cbb412744"} Nov 02 01:14:18.863866 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:18.863601 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074-original-pull-secret\") pod \"global-pull-secret-syncer-kl8v7\" (UID: \"9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074\") " pod="kube-system/global-pull-secret-syncer-kl8v7" Nov 02 01:14:18.864144 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:18.864121 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 01:14:18.864242 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:18.864206 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074-original-pull-secret podName:9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:20.864182547 +0000 UTC m=+6.070475211 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074-original-pull-secret") pod "global-pull-secret-syncer-kl8v7" (UID: "9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074") : object "kube-system"/"original-pull-secret" not registered Nov 02 01:14:18.965264 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:18.965222 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c7b5fa9-d0b9-4147-bc52-bdd444a2c081-metrics-certs\") pod \"network-metrics-daemon-nkqjm\" (UID: \"4c7b5fa9-d0b9-4147-bc52-bdd444a2c081\") " pod="openshift-multus/network-metrics-daemon-nkqjm" Nov 02 01:14:18.965444 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:18.965299 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxrsj\" (UniqueName: \"kubernetes.io/projected/3917d88c-88f5-4c5c-b1f7-e0195a61f88f-kube-api-access-hxrsj\") pod \"network-check-target-fdptg\" (UID: \"3917d88c-88f5-4c5c-b1f7-e0195a61f88f\") " pod="openshift-network-diagnostics/network-check-target-fdptg" Nov 02 01:14:18.965538 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:18.965515 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 01:14:18.965598 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:18.965544 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 01:14:18.965598 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:18.965559 2578 projected.go:196] Error preparing data for projected volume kube-api-access-hxrsj for pod openshift-network-diagnostics/network-check-target-fdptg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 01:14:18.965698 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:18.965625 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3917d88c-88f5-4c5c-b1f7-e0195a61f88f-kube-api-access-hxrsj podName:3917d88c-88f5-4c5c-b1f7-e0195a61f88f nodeName:}" failed. No retries permitted until 2026-11-02 01:14:20.965604825 +0000 UTC m=+6.171897491 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hxrsj" (UniqueName: "kubernetes.io/projected/3917d88c-88f5-4c5c-b1f7-e0195a61f88f-kube-api-access-hxrsj") pod "network-check-target-fdptg" (UID: "3917d88c-88f5-4c5c-b1f7-e0195a61f88f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 01:14:18.966103 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:18.966079 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 01:14:18.966172 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:18.966144 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c7b5fa9-d0b9-4147-bc52-bdd444a2c081-metrics-certs podName:4c7b5fa9-d0b9-4147-bc52-bdd444a2c081 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:20.966127434 +0000 UTC m=+6.172420090 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4c7b5fa9-d0b9-4147-bc52-bdd444a2c081-metrics-certs") pod "network-metrics-daemon-nkqjm" (UID: "4c7b5fa9-d0b9-4147-bc52-bdd444a2c081") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 01:14:19.278950 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:19.278850 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 01:09:17 +0000 UTC" deadline="2028-07-10 10:24:50.338046396 +0000 UTC" Nov 02 01:14:19.278950 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:19.278901 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14793h10m31.059150564s" Nov 02 01:14:19.363019 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:19.362964 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkqjm" Nov 02 01:14:19.363019 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:19.362999 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kl8v7" Nov 02 01:14:19.363252 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:19.363155 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkqjm" podUID="4c7b5fa9-d0b9-4147-bc52-bdd444a2c081" Nov 02 01:14:19.363522 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:19.363483 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kl8v7" podUID="9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074" Nov 02 01:14:19.452873 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:19.452833 2578 generic.go:356] "Generic (PLEG): container finished" podID="b96c82052e4f897819ee4482b034eaf4" containerID="129512017e109b75bcb5812581d26004e489e69f2b4523cf7117fd9021625a78" exitCode=0 Nov 02 01:14:19.453065 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:19.453033 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-130.us-east-2.compute.internal" event={"ID":"b96c82052e4f897819ee4482b034eaf4","Type":"ContainerDied","Data":"129512017e109b75bcb5812581d26004e489e69f2b4523cf7117fd9021625a78"} Nov 02 01:14:19.477405 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:19.476796 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-130.us-east-2.compute.internal" podStartSLOduration=2.476777015 podStartE2EDuration="2.476777015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:14:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:14:18.42015128 +0000 UTC m=+3.626443947" watchObservedRunningTime="2026-11-02 01:14:19.476777015 +0000 UTC m=+4.683069681" Nov 02 01:14:20.363102 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:20.363034 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fdptg" Nov 02 01:14:20.363742 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:20.363268 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fdptg" podUID="3917d88c-88f5-4c5c-b1f7-e0195a61f88f" Nov 02 01:14:20.467628 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:20.467587 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-130.us-east-2.compute.internal" event={"ID":"b96c82052e4f897819ee4482b034eaf4","Type":"ContainerStarted","Data":"1d12683f13340c1587b03fe580d5dd0be379c9b659bde5e6b9fbf1e4122d14ee"} Nov 02 01:14:20.883983 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:20.883946 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074-original-pull-secret\") pod \"global-pull-secret-syncer-kl8v7\" (UID: \"9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074\") " pod="kube-system/global-pull-secret-syncer-kl8v7" Nov 02 01:14:20.884158 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:20.884143 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 01:14:20.884219 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:20.884206 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074-original-pull-secret podName:9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:24.884188234 +0000 UTC m=+10.090480877 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074-original-pull-secret") pod "global-pull-secret-syncer-kl8v7" (UID: "9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074") : object "kube-system"/"original-pull-secret" not registered Nov 02 01:14:20.984880 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:20.984849 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxrsj\" (UniqueName: \"kubernetes.io/projected/3917d88c-88f5-4c5c-b1f7-e0195a61f88f-kube-api-access-hxrsj\") pod \"network-check-target-fdptg\" (UID: \"3917d88c-88f5-4c5c-b1f7-e0195a61f88f\") " pod="openshift-network-diagnostics/network-check-target-fdptg" Nov 02 01:14:20.985036 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:20.984952 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c7b5fa9-d0b9-4147-bc52-bdd444a2c081-metrics-certs\") pod \"network-metrics-daemon-nkqjm\" (UID: \"4c7b5fa9-d0b9-4147-bc52-bdd444a2c081\") " pod="openshift-multus/network-metrics-daemon-nkqjm" Nov 02 01:14:20.985126 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:20.985112 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 01:14:20.985178 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:20.985169 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c7b5fa9-d0b9-4147-bc52-bdd444a2c081-metrics-certs podName:4c7b5fa9-d0b9-4147-bc52-bdd444a2c081 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:24.985151331 +0000 UTC m=+10.191443986 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4c7b5fa9-d0b9-4147-bc52-bdd444a2c081-metrics-certs") pod "network-metrics-daemon-nkqjm" (UID: "4c7b5fa9-d0b9-4147-bc52-bdd444a2c081") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 01:14:20.985574 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:20.985553 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 01:14:20.985671 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:20.985577 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 01:14:20.985671 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:20.985589 2578 projected.go:196] Error preparing data for projected volume kube-api-access-hxrsj for pod openshift-network-diagnostics/network-check-target-fdptg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 01:14:20.985671 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:20.985629 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3917d88c-88f5-4c5c-b1f7-e0195a61f88f-kube-api-access-hxrsj podName:3917d88c-88f5-4c5c-b1f7-e0195a61f88f nodeName:}" failed. No retries permitted until 2026-11-02 01:14:24.985615762 +0000 UTC m=+10.191908425 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hxrsj" (UniqueName: "kubernetes.io/projected/3917d88c-88f5-4c5c-b1f7-e0195a61f88f-kube-api-access-hxrsj") pod "network-check-target-fdptg" (UID: "3917d88c-88f5-4c5c-b1f7-e0195a61f88f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 01:14:21.371711 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:21.371679 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkqjm" Nov 02 01:14:21.372229 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:21.371812 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkqjm" podUID="4c7b5fa9-d0b9-4147-bc52-bdd444a2c081" Nov 02 01:14:21.372229 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:21.372183 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kl8v7" Nov 02 01:14:21.372337 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:21.372275 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kl8v7" podUID="9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074" Nov 02 01:14:21.475933 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:21.474579 2578 generic.go:356] "Generic (PLEG): container finished" podID="6a02556d-4538-478d-84cf-a4d104565c84" containerID="d256879fb5ca93312bd104d0a61b5c405f0fe8bed03c321c0251d1626f524d66" exitCode=0 Nov 02 01:14:21.475933 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:21.475688 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-snk78" event={"ID":"6a02556d-4538-478d-84cf-a4d104565c84","Type":"ContainerDied","Data":"d256879fb5ca93312bd104d0a61b5c405f0fe8bed03c321c0251d1626f524d66"} Nov 02 01:14:21.502306 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:21.502252 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-130.us-east-2.compute.internal" podStartSLOduration=4.502234417 podStartE2EDuration="4.502234417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:14:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:14:20.486065663 +0000 UTC m=+5.692358326" watchObservedRunningTime="2026-11-02 01:14:21.502234417 +0000 UTC m=+6.708527080" Nov 02 01:14:22.363517 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:22.363482 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fdptg" Nov 02 01:14:22.363709 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:22.363638 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fdptg" podUID="3917d88c-88f5-4c5c-b1f7-e0195a61f88f" Nov 02 01:14:22.482351 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:22.481653 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vcsls" event={"ID":"16eed615-a448-4416-b06e-1c33d1fffb6d","Type":"ContainerStarted","Data":"ebc3ad951f7995121e0c8eaea5d9b9184f2a807bbe123da7e3a77e8552155f17"} Nov 02 01:14:23.367303 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:23.366511 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kl8v7" Nov 02 01:14:23.367303 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:23.366707 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kl8v7" podUID="9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074" Nov 02 01:14:23.367303 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:23.367070 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkqjm" Nov 02 01:14:23.367303 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:23.367180 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkqjm" podUID="4c7b5fa9-d0b9-4147-bc52-bdd444a2c081" Nov 02 01:14:23.490077 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:23.490037 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bgczm" event={"ID":"41f79a83-9886-4c59-a904-328463dfe674","Type":"ContainerStarted","Data":"1d0b3a4cd51cb0dba4da7ed8017da3ca9dc97c7d4c895215d196de3e220bd91a"} Nov 02 01:14:23.507227 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:23.506241 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bgczm" podStartSLOduration=3.297425293 podStartE2EDuration="8.506221449s" totalImagesPullingTime="5.208796156s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:14:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:14:23.505947324 +0000 UTC m=+8.712239987" watchObservedRunningTime="2026-11-02 01:14:23.506221449 +0000 UTC m=+8.712514114" Nov 02 01:14:23.507227 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:23.506497 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vcsls" podStartSLOduration=4.344096165 podStartE2EDuration="8.506490974s" totalImagesPullingTime="4.162394809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:14:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:14:22.500592691 +0000 UTC m=+7.706885355" watchObservedRunningTime="2026-11-02 01:14:23.506490974 +0000 UTC m=+8.712783637" Nov 02 01:14:24.364049 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:24.363551 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fdptg" Nov 02 01:14:24.364049 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:24.363684 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fdptg" podUID="3917d88c-88f5-4c5c-b1f7-e0195a61f88f" Nov 02 01:14:24.493061 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:24.492991 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj5fn" event={"ID":"20bc48c0-b0ef-40df-a5c9-06c48c75fe20","Type":"ContainerStarted","Data":"4fbcb7eda810e66f66bc09f52dd4709269f23c61556947a93ceeaa85fc9ede0e"} Nov 02 01:14:24.923669 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:24.923105 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074-original-pull-secret\") pod \"global-pull-secret-syncer-kl8v7\" (UID: \"9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074\") " pod="kube-system/global-pull-secret-syncer-kl8v7" Nov 02 01:14:24.923669 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:24.923238 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 01:14:24.923669 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:24.923301 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074-original-pull-secret podName:9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:32.923282809 +0000 UTC m=+18.129575469 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074-original-pull-secret") pod "global-pull-secret-syncer-kl8v7" (UID: "9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074") : object "kube-system"/"original-pull-secret" not registered Nov 02 01:14:25.026712 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:25.026035 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c7b5fa9-d0b9-4147-bc52-bdd444a2c081-metrics-certs\") pod \"network-metrics-daemon-nkqjm\" (UID: \"4c7b5fa9-d0b9-4147-bc52-bdd444a2c081\") " pod="openshift-multus/network-metrics-daemon-nkqjm" Nov 02 01:14:25.026712 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:25.026091 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxrsj\" (UniqueName: \"kubernetes.io/projected/3917d88c-88f5-4c5c-b1f7-e0195a61f88f-kube-api-access-hxrsj\") pod \"network-check-target-fdptg\" (UID: \"3917d88c-88f5-4c5c-b1f7-e0195a61f88f\") " pod="openshift-network-diagnostics/network-check-target-fdptg" Nov 02 01:14:25.026712 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:25.026222 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 01:14:25.026712 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:25.026238 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 01:14:25.026712 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:25.026249 2578 projected.go:196] Error preparing data for projected volume kube-api-access-hxrsj for pod openshift-network-diagnostics/network-check-target-fdptg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 01:14:25.026712 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:25.026299 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3917d88c-88f5-4c5c-b1f7-e0195a61f88f-kube-api-access-hxrsj podName:3917d88c-88f5-4c5c-b1f7-e0195a61f88f nodeName:}" failed. No retries permitted until 2026-11-02 01:14:33.026284136 +0000 UTC m=+18.232576790 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hxrsj" (UniqueName: "kubernetes.io/projected/3917d88c-88f5-4c5c-b1f7-e0195a61f88f-kube-api-access-hxrsj") pod "network-check-target-fdptg" (UID: "3917d88c-88f5-4c5c-b1f7-e0195a61f88f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 01:14:25.026712 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:25.026359 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 01:14:25.026712 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:25.026407 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c7b5fa9-d0b9-4147-bc52-bdd444a2c081-metrics-certs podName:4c7b5fa9-d0b9-4147-bc52-bdd444a2c081 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:33.026397558 +0000 UTC m=+18.232690211 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4c7b5fa9-d0b9-4147-bc52-bdd444a2c081-metrics-certs") pod "network-metrics-daemon-nkqjm" (UID: "4c7b5fa9-d0b9-4147-bc52-bdd444a2c081") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 01:14:25.365354 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:25.365324 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kl8v7" Nov 02 01:14:25.365517 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:25.365454 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkqjm" Nov 02 01:14:25.365517 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:25.365459 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kl8v7" podUID="9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074" Nov 02 01:14:25.365593 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:25.365535 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkqjm" podUID="4c7b5fa9-d0b9-4147-bc52-bdd444a2c081" Nov 02 01:14:25.779877 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:25.779844 2578 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 01:14:26.306054 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:26.305929 2578 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T01:14:25.77987036Z","UUID":"54d8ac2d-1e0a-4998-86ce-2bd1d63d06db","Handler":null,"Name":"","Endpoint":""} Nov 02 01:14:26.308002 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:26.307971 2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 01:14:26.308002 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:26.308006 2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 01:14:26.363637 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:26.363595 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fdptg" Nov 02 01:14:26.363784 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:26.363732 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fdptg" podUID="3917d88c-88f5-4c5c-b1f7-e0195a61f88f" Nov 02 01:14:26.503434 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:26.503399 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qwzn4" event={"ID":"77df90af-e163-47ea-83ad-99bc1258f3d5","Type":"ContainerStarted","Data":"84f8f0bb18402855390af72c5c42ae84dca7f49e1d4182ce2872d4e26e9060be"} Nov 02 01:14:26.506755 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:26.506723 2578 generic.go:356] "Generic (PLEG): container finished" podID="6a02556d-4538-478d-84cf-a4d104565c84" containerID="89bdd479f0394295448197347410602769e38605c6035ab1a86bcdcb2c5ffcd8" exitCode=0 Nov 02 01:14:26.506885 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:26.506819 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-snk78" event={"ID":"6a02556d-4538-478d-84cf-a4d104565c84","Type":"ContainerDied","Data":"89bdd479f0394295448197347410602769e38605c6035ab1a86bcdcb2c5ffcd8"} Nov 02 01:14:26.509525 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:26.509500 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj5fn" event={"ID":"20bc48c0-b0ef-40df-a5c9-06c48c75fe20","Type":"ContainerStarted","Data":"219e8a889e79b5391f5267e6d348844c6d6e6cd0bf374d2ee3ee41df2228d2e8"} Nov 02 01:14:26.511403 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:26.511333 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-27jtx" event={"ID":"4a12c42f-81b2-4fde-bc01-8abe3e640957","Type":"ContainerStarted","Data":"26c86d83f5c5d236b21a421b93133af44daa2848039d52902ea17b4c8eab1b7d"} Nov 02 01:14:26.521494 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:26.521451 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-qwzn4" podStartSLOduration=3.183542203 podStartE2EDuration="11.521431828s" totalImagesPullingTime="8.337889625s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:14:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:14:26.520736952 +0000 UTC m=+11.727029613" watchObservedRunningTime="2026-11-02 01:14:26.521431828 +0000 UTC m=+11.727724491" Nov 02 01:14:26.541812 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:26.541756 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-27jtx" podStartSLOduration=3.616146841 podStartE2EDuration="11.541742218s" totalImagesPullingTime="7.925595377s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:14:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:14:26.540829445 +0000 UTC m=+11.747122111" watchObservedRunningTime="2026-11-02 01:14:26.541742218 +0000 UTC m=+11.748034880" Nov 02 01:14:26.889187 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:26.888569 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vcsls" Nov 02 01:14:26.890123 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:26.890100 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vcsls" Nov 02 01:14:27.363528 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:27.363321 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kl8v7" Nov 02 01:14:27.363712 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:27.363633 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kl8v7" podUID="9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074" Nov 02 01:14:27.363822 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:27.363344 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkqjm" Nov 02 01:14:27.363935 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:27.363913 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkqjm" podUID="4c7b5fa9-d0b9-4147-bc52-bdd444a2c081" Nov 02 01:14:27.517909 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:27.517871 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj5fn" event={"ID":"20bc48c0-b0ef-40df-a5c9-06c48c75fe20","Type":"ContainerStarted","Data":"0a1c73b645246357fc8e2b07a95ec5711d6b621195020f02b1a92b368aa27b3f"} Nov 02 01:14:27.519783 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:27.519701 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-phtn9" event={"ID":"497c02e6-eb5a-49ed-a911-97384fb409cf","Type":"ContainerStarted","Data":"8da608b2b9062c1f4d949108aad45fe890d9b74557997a7d568a1b5fe536590b"} Nov 02 01:14:27.538766 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:27.538713 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj5fn" podStartSLOduration=3.578736101 podStartE2EDuration="12.538695957s" totalImagesPullingTime="8.959959856s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:14:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:14:27.53792633 +0000 UTC m=+12.744218992" watchObservedRunningTime="2026-11-02 01:14:27.538695957 +0000 UTC m=+12.744988623" Nov 02 01:14:27.553748 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:27.553704 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-phtn9" podStartSLOduration=4.327190241 podStartE2EDuration="12.5536899s" totalImagesPullingTime="8.226499659s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:14:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:14:27.553618228 +0000 UTC m=+12.759910892" watchObservedRunningTime="2026-11-02 01:14:27.5536899 +0000 UTC m=+12.759982752" Nov 02 01:14:28.362693 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:28.362667 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fdptg" Nov 02 01:14:28.363082 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:28.362786 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fdptg" podUID="3917d88c-88f5-4c5c-b1f7-e0195a61f88f" Nov 02 01:14:28.523247 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:28.523209 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" event={"ID":"8cd0f6e2-e491-4b6c-943a-c1115300dcaf","Type":"ContainerStarted","Data":"cc4fd1fc2a3df0222c7a13e33f10ad0244f7cf3375f9b988512190d23d88a8fe"} Nov 02 01:14:28.525299 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:28.525273 2578 generic.go:356] "Generic (PLEG): container finished" podID="6a02556d-4538-478d-84cf-a4d104565c84" containerID="bfa0e14112d985d99c151707104fd44b4f23b2480698fadfd4d23375f951165e" exitCode=0 Nov 02 01:14:28.525441 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:28.525357 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-snk78" event={"ID":"6a02556d-4538-478d-84cf-a4d104565c84","Type":"ContainerDied","Data":"bfa0e14112d985d99c151707104fd44b4f23b2480698fadfd4d23375f951165e"} Nov 02 01:14:28.545115 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:28.545074 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" podStartSLOduration=2.828991742 podStartE2EDuration="13.545061054s" totalImagesPullingTime="10.716069312s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:14:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:14:28.544920713 +0000 UTC m=+13.751213376" watchObservedRunningTime="2026-11-02 01:14:28.545061054 +0000 UTC m=+13.751353716" Nov 02 01:14:29.363011 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:29.362976 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kl8v7" Nov 02 01:14:29.363666 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:29.363139 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kl8v7" podUID="9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074" Nov 02 01:14:29.363666 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:29.363589 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkqjm" Nov 02 01:14:29.364192 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:29.364168 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkqjm" podUID="4c7b5fa9-d0b9-4147-bc52-bdd444a2c081" Nov 02 01:14:30.363131 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:30.363089 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fdptg" Nov 02 01:14:30.363893 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:30.363222 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fdptg" podUID="3917d88c-88f5-4c5c-b1f7-e0195a61f88f" Nov 02 01:14:30.535779 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:30.535733 2578 generic.go:356] "Generic (PLEG): container finished" podID="6a02556d-4538-478d-84cf-a4d104565c84" containerID="6807bcba93370507b7b9c14d68ae9269196d027dc71ac2b2d16b2bc93aa10005" exitCode=0 Nov 02 01:14:30.535779 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:30.535776 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-snk78" event={"ID":"6a02556d-4538-478d-84cf-a4d104565c84","Type":"ContainerDied","Data":"6807bcba93370507b7b9c14d68ae9269196d027dc71ac2b2d16b2bc93aa10005"} Nov 02 01:14:31.363465 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:31.363429 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kl8v7" Nov 02 01:14:31.363934 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:31.363469 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkqjm" Nov 02 01:14:31.363934 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:31.363575 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kl8v7" podUID="9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074" Nov 02 01:14:31.363934 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:31.363720 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkqjm" podUID="4c7b5fa9-d0b9-4147-bc52-bdd444a2c081" Nov 02 01:14:32.362953 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:32.362767 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fdptg" Nov 02 01:14:32.363205 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:32.363046 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fdptg" podUID="3917d88c-88f5-4c5c-b1f7-e0195a61f88f" Nov 02 01:14:32.988595 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:32.988569 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074-original-pull-secret\") pod \"global-pull-secret-syncer-kl8v7\" (UID: \"9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074\") " pod="kube-system/global-pull-secret-syncer-kl8v7" Nov 02 01:14:32.988984 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:32.988721 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 01:14:32.988984 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:32.988791 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074-original-pull-secret podName:9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:48.988774255 +0000 UTC m=+34.195066912 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074-original-pull-secret") pod "global-pull-secret-syncer-kl8v7" (UID: "9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074") : object "kube-system"/"original-pull-secret" not registered Nov 02 01:14:33.089226 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:33.089198 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxrsj\" (UniqueName: \"kubernetes.io/projected/3917d88c-88f5-4c5c-b1f7-e0195a61f88f-kube-api-access-hxrsj\") pod \"network-check-target-fdptg\" (UID: \"3917d88c-88f5-4c5c-b1f7-e0195a61f88f\") " pod="openshift-network-diagnostics/network-check-target-fdptg" Nov 02 01:14:33.089331 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:33.089289 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c7b5fa9-d0b9-4147-bc52-bdd444a2c081-metrics-certs\") pod \"network-metrics-daemon-nkqjm\" (UID: \"4c7b5fa9-d0b9-4147-bc52-bdd444a2c081\") " pod="openshift-multus/network-metrics-daemon-nkqjm" Nov 02 01:14:33.089404 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:33.089357 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 01:14:33.089404 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:33.089398 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 01:14:33.089512 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:33.089414 2578 projected.go:196] Error preparing data for projected volume kube-api-access-hxrsj for pod openshift-network-diagnostics/network-check-target-fdptg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 01:14:33.089512 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:33.089427 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 01:14:33.089512 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:33.089460 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3917d88c-88f5-4c5c-b1f7-e0195a61f88f-kube-api-access-hxrsj podName:3917d88c-88f5-4c5c-b1f7-e0195a61f88f nodeName:}" failed. No retries permitted until 2026-11-02 01:14:49.089446395 +0000 UTC m=+34.295739055 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hxrsj" (UniqueName: "kubernetes.io/projected/3917d88c-88f5-4c5c-b1f7-e0195a61f88f-kube-api-access-hxrsj") pod "network-check-target-fdptg" (UID: "3917d88c-88f5-4c5c-b1f7-e0195a61f88f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 01:14:33.089512 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:33.089484 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c7b5fa9-d0b9-4147-bc52-bdd444a2c081-metrics-certs podName:4c7b5fa9-d0b9-4147-bc52-bdd444a2c081 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:49.089467717 +0000 UTC m=+34.295760382 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4c7b5fa9-d0b9-4147-bc52-bdd444a2c081-metrics-certs") pod "network-metrics-daemon-nkqjm" (UID: "4c7b5fa9-d0b9-4147-bc52-bdd444a2c081") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 01:14:33.365637 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:33.365612 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkqjm" Nov 02 01:14:33.365783 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:33.365707 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkqjm" podUID="4c7b5fa9-d0b9-4147-bc52-bdd444a2c081" Nov 02 01:14:33.365783 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:33.365765 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kl8v7" Nov 02 01:14:33.365865 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:33.365847 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kl8v7" podUID="9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074" Nov 02 01:14:33.545073 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:33.545038 2578 generic.go:356] "Generic (PLEG): container finished" podID="6a02556d-4538-478d-84cf-a4d104565c84" containerID="2d41b2e6fc23f326705993f0fe16ed0a0bca86edd1030b74691bf2e5f3968f81" exitCode=0 Nov 02 01:14:33.545236 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:33.545118 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-snk78" event={"ID":"6a02556d-4538-478d-84cf-a4d104565c84","Type":"ContainerDied","Data":"2d41b2e6fc23f326705993f0fe16ed0a0bca86edd1030b74691bf2e5f3968f81"} Nov 02 01:14:33.548368 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:33.548339 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" event={"ID":"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7","Type":"ContainerStarted","Data":"bd6d6cc6f8ed9c0b41b5282d308ac25f57d0a451fa2450da30e7bbab15fdc4ca"} Nov 02 01:14:33.548502 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:33.548437 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" event={"ID":"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7","Type":"ContainerStarted","Data":"6421ac44f0f747a6f750b17111bb53582ad685109fe4cb9b3524b86863929be3"} Nov 02 01:14:33.548502 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:33.548487 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" event={"ID":"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7","Type":"ContainerStarted","Data":"7d1968257505576b416e7208ace2996e2a9d9fe358f2ef0045f048904fa24d35"} Nov 02 01:14:33.548614 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:33.548534 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" event={"ID":"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7","Type":"ContainerStarted","Data":"0f658a6d28734f17386e474ebb81875d6cd187cc09517f65a285543986e58d70"} Nov 02 01:14:33.548665 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:33.548625 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" event={"ID":"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7","Type":"ContainerStarted","Data":"ec404a9dc6217f0ed1856e511f6853ff1d5bf503ee78202f7b3c6afad3b8c4b0"} Nov 02 01:14:33.548717 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:33.548674 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" event={"ID":"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7","Type":"ContainerStarted","Data":"eba83f4f72051dfcd60500e09153f5f737e3cb03a733d0f702cd82d3a8c90529"} Nov 02 01:14:34.363061 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:34.363026 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fdptg" Nov 02 01:14:34.363691 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:34.363131 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fdptg" podUID="3917d88c-88f5-4c5c-b1f7-e0195a61f88f" Nov 02 01:14:34.552991 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:34.552956 2578 generic.go:356] "Generic (PLEG): container finished" podID="6a02556d-4538-478d-84cf-a4d104565c84" containerID="1c556fc4305e0e0a9fa951050f19cb59acd55fc59ff711195dd43234dce73b0d" exitCode=0 Nov 02 01:14:34.553126 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:34.553012 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-snk78" event={"ID":"6a02556d-4538-478d-84cf-a4d104565c84","Type":"ContainerDied","Data":"1c556fc4305e0e0a9fa951050f19cb59acd55fc59ff711195dd43234dce73b0d"} Nov 02 01:14:35.366083 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:35.366051 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkqjm" Nov 02 01:14:35.366464 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:35.366153 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkqjm" podUID="4c7b5fa9-d0b9-4147-bc52-bdd444a2c081" Nov 02 01:14:35.366464 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:35.366179 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kl8v7" Nov 02 01:14:35.366464 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:35.366226 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kl8v7" podUID="9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074" Nov 02 01:14:35.559112 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:35.559028 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-snk78" event={"ID":"6a02556d-4538-478d-84cf-a4d104565c84","Type":"ContainerStarted","Data":"8e028ab947d0ce411f8affeb739cef0ca281d1996e9f728b30812286e17da8bf"} Nov 02 01:14:35.562220 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:35.562194 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" event={"ID":"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7","Type":"ContainerStarted","Data":"1f0139c18f4625a424fd27c8005af041cf80a7dbfbf5f773bb5d879301dcaee9"} Nov 02 01:14:35.586777 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:35.586737 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-snk78" podStartSLOduration=8.163269297 podStartE2EDuration="20.58672488s" totalImagesPullingTime="12.423455583s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:14:15 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:14:35.586219435 +0000 UTC m=+20.792512096" watchObservedRunningTime="2026-11-02 01:14:35.58672488 +0000 UTC m=+20.793017541" Nov 02 01:14:36.363661 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:36.363625 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fdptg" Nov 02 01:14:36.363819 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:36.363732 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fdptg" podUID="3917d88c-88f5-4c5c-b1f7-e0195a61f88f" Nov 02 01:14:37.363347 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:37.363315 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kl8v7" Nov 02 01:14:37.363978 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:37.363474 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kl8v7" podUID="9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074" Nov 02 01:14:37.363978 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:37.363545 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkqjm" Nov 02 01:14:37.363978 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:37.363631 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkqjm" podUID="4c7b5fa9-d0b9-4147-bc52-bdd444a2c081" Nov 02 01:14:37.569909 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:37.569873 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" event={"ID":"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7","Type":"ContainerStarted","Data":"35895cf9d70dc20cd4da7b5ce4127387e5f3883cab31b7ec14e13f422c2e0b5d"} Nov 02 01:14:37.570164 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:37.570127 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:37.570275 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:37.570256 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:37.587720 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:37.587500 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:37.587841 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:37.587759 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:37.607659 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:37.607612 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" podStartSLOduration=7.405532366 podStartE2EDuration="22.607600194s" totalImagesPullingTime="15.202067828s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:14:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:14:37.606909536 +0000 UTC m=+22.813202199" watchObservedRunningTime="2026-11-02 01:14:37.607600194 +0000 UTC m=+22.813892852" Nov 02 01:14:38.363460 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:38.363426 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fdptg" Nov 02 01:14:38.363854 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:38.363534 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fdptg" podUID="3917d88c-88f5-4c5c-b1f7-e0195a61f88f" Nov 02 01:14:38.572511 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:38.572470 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 01:14:39.120775 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:39.120732 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kl8v7"] Nov 02 01:14:39.120968 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:39.120875 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kl8v7" Nov 02 01:14:39.121049 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:39.121021 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kl8v7" podUID="9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074" Nov 02 01:14:39.121313 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:39.121297 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fdptg"] Nov 02 01:14:39.121497 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:39.121482 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fdptg" Nov 02 01:14:39.121595 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:39.121576 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fdptg" podUID="3917d88c-88f5-4c5c-b1f7-e0195a61f88f" Nov 02 01:14:39.121929 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:39.121907 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nkqjm"] Nov 02 01:14:39.122018 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:39.122005 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkqjm" Nov 02 01:14:39.122111 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:39.122088 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkqjm" podUID="4c7b5fa9-d0b9-4147-bc52-bdd444a2c081" Nov 02 01:14:39.574482 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:39.574404 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 01:14:39.657686 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:39.657660 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:40.362946 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:40.362890 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fdptg" Nov 02 01:14:40.363133 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:40.363016 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fdptg" podUID="3917d88c-88f5-4c5c-b1f7-e0195a61f88f" Nov 02 01:14:41.365939 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:41.365909 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kl8v7" Nov 02 01:14:41.366486 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:41.366020 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kl8v7" podUID="9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074" Nov 02 01:14:41.366486 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:41.366069 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkqjm" Nov 02 01:14:41.366486 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:41.366168 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkqjm" podUID="4c7b5fa9-d0b9-4147-bc52-bdd444a2c081" Nov 02 01:14:42.363330 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:42.363294 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fdptg" Nov 02 01:14:42.363489 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:42.363437 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fdptg" podUID="3917d88c-88f5-4c5c-b1f7-e0195a61f88f" Nov 02 01:14:43.366339 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:43.366303 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkqjm" Nov 02 01:14:43.366803 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:43.366451 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkqjm" podUID="4c7b5fa9-d0b9-4147-bc52-bdd444a2c081" Nov 02 01:14:43.366803 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:43.366502 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kl8v7" Nov 02 01:14:43.366803 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:43.366590 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kl8v7" podUID="9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074" Nov 02 01:14:44.363623 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:44.363590 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fdptg" Nov 02 01:14:44.363820 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:44.363688 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fdptg" podUID="3917d88c-88f5-4c5c-b1f7-e0195a61f88f" Nov 02 01:14:45.142856 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:45.142823 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-130.us-east-2.compute.internal" event="NodeReady" Nov 02 01:14:45.143329 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:45.142973 2578 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 02 01:14:45.193204 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:45.193116 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-pn4kl"] Nov 02 01:14:45.198557 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:45.198539 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xcm6b"] Nov 02 01:14:45.198736 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:45.198715 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pn4kl" Nov 02 01:14:45.201956 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:45.201934 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xcm6b" Nov 02 01:14:45.205683 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:45.205662 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 01:14:45.205683 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:45.205673 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 01:14:45.207133 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:45.207112 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pn4kl"] Nov 02 01:14:45.207788 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:45.207768 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xcm6b"] Nov 02 01:14:45.209012 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:45.208989 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 01:14:45.209122 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:45.209104 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 01:14:45.209698 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:45.209679 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-rcvqj\"" Nov 02 01:14:45.209863 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:45.209845 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-g6rtc\"" Nov 02 01:14:45.209943 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:45.209930 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 01:14:45.366198 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:45.366162 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkqjm" Nov 02 01:14:45.366198 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:45.366198 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kl8v7" Nov 02 01:14:45.370414 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:45.370391 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 01:14:45.370599 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:45.370585 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wfrhp\"" Nov 02 01:14:45.370661 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:45.370635 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-7fmpb\"" Nov 02 01:14:45.370661 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:45.370636 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 01:14:45.372137 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:45.372118 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a75a9f45-7678-4014-92d9-93d3f5b260ba-cert\") pod \"ingress-canary-pn4kl\" (UID: \"a75a9f45-7678-4014-92d9-93d3f5b260ba\") " pod="openshift-ingress-canary/ingress-canary-pn4kl" Nov 02 01:14:45.372253 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:45.372148 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9pbx\" (UniqueName: \"kubernetes.io/projected/00d17dbb-cb72-4413-9559-4487c313158c-kube-api-access-g9pbx\") pod \"dns-default-xcm6b\" (UID: \"00d17dbb-cb72-4413-9559-4487c313158c\") " pod="openshift-dns/dns-default-xcm6b" Nov 02 01:14:45.372321 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:45.372256 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/00d17dbb-cb72-4413-9559-4487c313158c-metrics-tls\") pod \"dns-default-xcm6b\" (UID: \"00d17dbb-cb72-4413-9559-4487c313158c\") " pod="openshift-dns/dns-default-xcm6b" Nov 02 01:14:45.372321 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:45.372296 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-987n6\" (UniqueName: \"kubernetes.io/projected/a75a9f45-7678-4014-92d9-93d3f5b260ba-kube-api-access-987n6\") pod \"ingress-canary-pn4kl\" (UID: \"a75a9f45-7678-4014-92d9-93d3f5b260ba\") " pod="openshift-ingress-canary/ingress-canary-pn4kl" Nov 02 01:14:45.372445 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:45.372348 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/00d17dbb-cb72-4413-9559-4487c313158c-config-volume\") pod \"dns-default-xcm6b\" (UID: \"00d17dbb-cb72-4413-9559-4487c313158c\") " pod="openshift-dns/dns-default-xcm6b" Nov 02 01:14:45.372445 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:45.372391 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00d17dbb-cb72-4413-9559-4487c313158c-tmp-dir\") pod \"dns-default-xcm6b\" (UID: \"00d17dbb-cb72-4413-9559-4487c313158c\") " pod="openshift-dns/dns-default-xcm6b" Nov 02 01:14:45.373172 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:45.373154 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vcsls" Nov 02 01:14:45.373254 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:45.373245 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 01:14:45.373704 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:45.373686 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vcsls" Nov 02 01:14:45.473001 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:45.472925 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9pbx\" (UniqueName: \"kubernetes.io/projected/00d17dbb-cb72-4413-9559-4487c313158c-kube-api-access-g9pbx\") pod \"dns-default-xcm6b\" (UID: \"00d17dbb-cb72-4413-9559-4487c313158c\") " pod="openshift-dns/dns-default-xcm6b" Nov 02 01:14:45.473156 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:45.473015 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/00d17dbb-cb72-4413-9559-4487c313158c-metrics-tls\") pod \"dns-default-xcm6b\" (UID: \"00d17dbb-cb72-4413-9559-4487c313158c\") " pod="openshift-dns/dns-default-xcm6b" Nov 02 01:14:45.473156 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:45.473102 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 01:14:45.473156 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:45.473154 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00d17dbb-cb72-4413-9559-4487c313158c-metrics-tls podName:00d17dbb-cb72-4413-9559-4487c313158c nodeName:}" failed. No retries permitted until 2026-11-02 01:14:45.973137227 +0000 UTC m=+31.179429889 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/00d17dbb-cb72-4413-9559-4487c313158c-metrics-tls") pod "dns-default-xcm6b" (UID: "00d17dbb-cb72-4413-9559-4487c313158c") : secret "dns-default-metrics-tls" not found Nov 02 01:14:45.473308 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:45.473173 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-987n6\" (UniqueName: \"kubernetes.io/projected/a75a9f45-7678-4014-92d9-93d3f5b260ba-kube-api-access-987n6\") pod \"ingress-canary-pn4kl\" (UID: \"a75a9f45-7678-4014-92d9-93d3f5b260ba\") " pod="openshift-ingress-canary/ingress-canary-pn4kl" Nov 02 01:14:45.473308 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:45.473210 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/00d17dbb-cb72-4413-9559-4487c313158c-config-volume\") pod \"dns-default-xcm6b\" (UID: \"00d17dbb-cb72-4413-9559-4487c313158c\") " pod="openshift-dns/dns-default-xcm6b" Nov 02 01:14:45.473308 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:45.473232 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00d17dbb-cb72-4413-9559-4487c313158c-tmp-dir\") pod \"dns-default-xcm6b\" (UID: \"00d17dbb-cb72-4413-9559-4487c313158c\") " pod="openshift-dns/dns-default-xcm6b" Nov 02 01:14:45.473497 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:45.473267 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a75a9f45-7678-4014-92d9-93d3f5b260ba-cert\") pod \"ingress-canary-pn4kl\" (UID: \"a75a9f45-7678-4014-92d9-93d3f5b260ba\") " pod="openshift-ingress-canary/ingress-canary-pn4kl" Nov 02 01:14:45.473850 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:45.473826 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00d17dbb-cb72-4413-9559-4487c313158c-tmp-dir\") pod \"dns-default-xcm6b\" (UID: \"00d17dbb-cb72-4413-9559-4487c313158c\") " pod="openshift-dns/dns-default-xcm6b" Nov 02 01:14:45.474011 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:45.473991 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/00d17dbb-cb72-4413-9559-4487c313158c-config-volume\") pod \"dns-default-xcm6b\" (UID: \"00d17dbb-cb72-4413-9559-4487c313158c\") " pod="openshift-dns/dns-default-xcm6b" Nov 02 01:14:45.477599 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:45.477575 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a75a9f45-7678-4014-92d9-93d3f5b260ba-cert\") pod \"ingress-canary-pn4kl\" (UID: \"a75a9f45-7678-4014-92d9-93d3f5b260ba\") " pod="openshift-ingress-canary/ingress-canary-pn4kl" Nov 02 01:14:45.485822 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:45.485794 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9pbx\" (UniqueName: \"kubernetes.io/projected/00d17dbb-cb72-4413-9559-4487c313158c-kube-api-access-g9pbx\") pod \"dns-default-xcm6b\" (UID: \"00d17dbb-cb72-4413-9559-4487c313158c\") " pod="openshift-dns/dns-default-xcm6b" Nov 02 01:14:45.486013 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:45.485982 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-987n6\" (UniqueName: \"kubernetes.io/projected/a75a9f45-7678-4014-92d9-93d3f5b260ba-kube-api-access-987n6\") pod \"ingress-canary-pn4kl\" (UID: \"a75a9f45-7678-4014-92d9-93d3f5b260ba\") " pod="openshift-ingress-canary/ingress-canary-pn4kl" Nov 02 01:14:45.519607 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:45.519575 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pn4kl" Nov 02 01:14:45.847485 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:45.847406 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pn4kl"] Nov 02 01:14:45.851272 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:45.851245 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda75a9f45_7678_4014_92d9_93d3f5b260ba.slice/crio-d434fbb46acfed32d19ce5ef28f90740f0cefe4f832dc4020078f7c956899364 WatchSource:0}: Error finding container d434fbb46acfed32d19ce5ef28f90740f0cefe4f832dc4020078f7c956899364: Status 404 returned error can't find the container with id d434fbb46acfed32d19ce5ef28f90740f0cefe4f832dc4020078f7c956899364 Nov 02 01:14:45.977794 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:45.977755 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/00d17dbb-cb72-4413-9559-4487c313158c-metrics-tls\") pod \"dns-default-xcm6b\" (UID: \"00d17dbb-cb72-4413-9559-4487c313158c\") " pod="openshift-dns/dns-default-xcm6b" Nov 02 01:14:45.977955 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:45.977900 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 01:14:45.977992 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:45.977959 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00d17dbb-cb72-4413-9559-4487c313158c-metrics-tls podName:00d17dbb-cb72-4413-9559-4487c313158c nodeName:}" failed. No retries permitted until 2026-11-02 01:14:46.977945398 +0000 UTC m=+32.184238038 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/00d17dbb-cb72-4413-9559-4487c313158c-metrics-tls") pod "dns-default-xcm6b" (UID: "00d17dbb-cb72-4413-9559-4487c313158c") : secret "dns-default-metrics-tls" not found Nov 02 01:14:46.363131 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:46.363091 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fdptg" Nov 02 01:14:46.367212 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:46.367190 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 01:14:46.367361 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:46.367190 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-g7hgp\"" Nov 02 01:14:46.367361 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:46.367232 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 01:14:46.591434 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:46.591400 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pn4kl" event={"ID":"a75a9f45-7678-4014-92d9-93d3f5b260ba","Type":"ContainerStarted","Data":"d434fbb46acfed32d19ce5ef28f90740f0cefe4f832dc4020078f7c956899364"} Nov 02 01:14:46.984432 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:46.984396 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/00d17dbb-cb72-4413-9559-4487c313158c-metrics-tls\") pod \"dns-default-xcm6b\" (UID: \"00d17dbb-cb72-4413-9559-4487c313158c\") " pod="openshift-dns/dns-default-xcm6b" Nov 02 01:14:46.984595 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:46.984539 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 01:14:46.984639 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:46.984602 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00d17dbb-cb72-4413-9559-4487c313158c-metrics-tls podName:00d17dbb-cb72-4413-9559-4487c313158c nodeName:}" failed. No retries permitted until 2026-11-02 01:14:48.984587772 +0000 UTC m=+34.190880436 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/00d17dbb-cb72-4413-9559-4487c313158c-metrics-tls") pod "dns-default-xcm6b" (UID: "00d17dbb-cb72-4413-9559-4487c313158c") : secret "dns-default-metrics-tls" not found Nov 02 01:14:48.998636 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:48.998568 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074-original-pull-secret\") pod \"global-pull-secret-syncer-kl8v7\" (UID: \"9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074\") " pod="kube-system/global-pull-secret-syncer-kl8v7" Nov 02 01:14:48.998636 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:48.998626 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/00d17dbb-cb72-4413-9559-4487c313158c-metrics-tls\") pod \"dns-default-xcm6b\" (UID: \"00d17dbb-cb72-4413-9559-4487c313158c\") " pod="openshift-dns/dns-default-xcm6b" Nov 02 01:14:48.999102 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:48.998753 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 01:14:48.999102 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:48.998813 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00d17dbb-cb72-4413-9559-4487c313158c-metrics-tls podName:00d17dbb-cb72-4413-9559-4487c313158c nodeName:}" failed. No retries permitted until 2026-11-02 01:14:52.998795509 +0000 UTC m=+38.205088161 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/00d17dbb-cb72-4413-9559-4487c313158c-metrics-tls") pod "dns-default-xcm6b" (UID: "00d17dbb-cb72-4413-9559-4487c313158c") : secret "dns-default-metrics-tls" not found Nov 02 01:14:49.001130 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:49.001107 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074-original-pull-secret\") pod \"global-pull-secret-syncer-kl8v7\" (UID: \"9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074\") " pod="kube-system/global-pull-secret-syncer-kl8v7" Nov 02 01:14:49.098929 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:49.098887 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c7b5fa9-d0b9-4147-bc52-bdd444a2c081-metrics-certs\") pod \"network-metrics-daemon-nkqjm\" (UID: \"4c7b5fa9-d0b9-4147-bc52-bdd444a2c081\") " pod="openshift-multus/network-metrics-daemon-nkqjm" Nov 02 01:14:49.099097 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:49.098945 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxrsj\" (UniqueName: \"kubernetes.io/projected/3917d88c-88f5-4c5c-b1f7-e0195a61f88f-kube-api-access-hxrsj\") pod \"network-check-target-fdptg\" (UID: \"3917d88c-88f5-4c5c-b1f7-e0195a61f88f\") " pod="openshift-network-diagnostics/network-check-target-fdptg" Nov 02 01:14:49.099097 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:49.099041 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 01:14:49.099174 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:49.099108 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c7b5fa9-d0b9-4147-bc52-bdd444a2c081-metrics-certs podName:4c7b5fa9-d0b9-4147-bc52-bdd444a2c081 nodeName:}" failed. No retries permitted until 2026-11-02 01:15:21.099092219 +0000 UTC m=+66.305384889 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4c7b5fa9-d0b9-4147-bc52-bdd444a2c081-metrics-certs") pod "network-metrics-daemon-nkqjm" (UID: "4c7b5fa9-d0b9-4147-bc52-bdd444a2c081") : secret "metrics-daemon-secret" not found Nov 02 01:14:49.101451 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:49.101433 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxrsj\" (UniqueName: \"kubernetes.io/projected/3917d88c-88f5-4c5c-b1f7-e0195a61f88f-kube-api-access-hxrsj\") pod \"network-check-target-fdptg\" (UID: \"3917d88c-88f5-4c5c-b1f7-e0195a61f88f\") " pod="openshift-network-diagnostics/network-check-target-fdptg" Nov 02 01:14:49.121109 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:49.121089 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fdptg" Nov 02 01:14:49.287853 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:49.287820 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kl8v7" Nov 02 01:14:49.472590 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:49.472560 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fdptg"] Nov 02 01:14:49.475309 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:49.475287 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3917d88c_88f5_4c5c_b1f7_e0195a61f88f.slice/crio-ff3aac8d819d03f31e61d35bc7b0dbfdfa90eb9e23d42f4aeabc88adb6fa7980 WatchSource:0}: Error finding container ff3aac8d819d03f31e61d35bc7b0dbfdfa90eb9e23d42f4aeabc88adb6fa7980: Status 404 returned error can't find the container with id ff3aac8d819d03f31e61d35bc7b0dbfdfa90eb9e23d42f4aeabc88adb6fa7980 Nov 02 01:14:49.597545 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:49.597513 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kl8v7"] Nov 02 01:14:49.602061 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:49.599073 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fdptg" event={"ID":"3917d88c-88f5-4c5c-b1f7-e0195a61f88f","Type":"ContainerStarted","Data":"ff3aac8d819d03f31e61d35bc7b0dbfdfa90eb9e23d42f4aeabc88adb6fa7980"} Nov 02 01:14:49.602453 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:49.602117 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9916e33f_7f1d_4cb3_81fc_f4eb8e2bf074.slice/crio-f04282285db6ef418877646fb9d613c41772a44089e6d1f1c462cf9d93d4cfa5 WatchSource:0}: Error finding container f04282285db6ef418877646fb9d613c41772a44089e6d1f1c462cf9d93d4cfa5: Status 404 returned error can't find the container with id f04282285db6ef418877646fb9d613c41772a44089e6d1f1c462cf9d93d4cfa5 Nov 02 01:14:50.602959 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:50.602899 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kl8v7" event={"ID":"9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074","Type":"ContainerStarted","Data":"f04282285db6ef418877646fb9d613c41772a44089e6d1f1c462cf9d93d4cfa5"} Nov 02 01:14:51.405201 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:51.405160 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-gs62j"] Nov 02 01:14:51.421954 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:51.421923 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gs62j"] Nov 02 01:14:51.422158 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:51.422103 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gs62j" Nov 02 01:14:51.431970 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:51.431946 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 01:14:51.432118 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:51.432092 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hdjbk\"" Nov 02 01:14:51.473360 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:51.473334 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 01:14:51.473556 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:51.473333 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 01:14:51.473556 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:51.473532 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 01:14:51.516170 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:51.516137 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntxrh\" (UniqueName: \"kubernetes.io/projected/23bf37a9-9b32-47f9-a4e0-364a7273c02f-kube-api-access-ntxrh\") pod \"insights-runtime-extractor-gs62j\" (UID: \"23bf37a9-9b32-47f9-a4e0-364a7273c02f\") " pod="openshift-insights/insights-runtime-extractor-gs62j" Nov 02 01:14:51.516334 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:51.516209 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/23bf37a9-9b32-47f9-a4e0-364a7273c02f-crio-socket\") pod \"insights-runtime-extractor-gs62j\" (UID: \"23bf37a9-9b32-47f9-a4e0-364a7273c02f\") " pod="openshift-insights/insights-runtime-extractor-gs62j" Nov 02 01:14:51.516334 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:51.516303 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/23bf37a9-9b32-47f9-a4e0-364a7273c02f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gs62j\" (UID: \"23bf37a9-9b32-47f9-a4e0-364a7273c02f\") " pod="openshift-insights/insights-runtime-extractor-gs62j" Nov 02 01:14:51.516509 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:51.516341 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/23bf37a9-9b32-47f9-a4e0-364a7273c02f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gs62j\" (UID: \"23bf37a9-9b32-47f9-a4e0-364a7273c02f\") " pod="openshift-insights/insights-runtime-extractor-gs62j" Nov 02 01:14:51.516564 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:51.516507 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/23bf37a9-9b32-47f9-a4e0-364a7273c02f-data-volume\") pod \"insights-runtime-extractor-gs62j\" (UID: \"23bf37a9-9b32-47f9-a4e0-364a7273c02f\") " pod="openshift-insights/insights-runtime-extractor-gs62j" Nov 02 01:14:51.553549 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:51.553508 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-ctstx"] Nov 02 01:14:51.569833 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:51.569805 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-ctstx"] Nov 02 01:14:51.570052 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:51.569927 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ctstx" Nov 02 01:14:51.580479 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:51.579600 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 02 01:14:51.580479 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:51.580158 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-ck74h\"" Nov 02 01:14:51.580479 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:51.580242 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 02 01:14:51.608693 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:51.608624 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pn4kl" event={"ID":"a75a9f45-7678-4014-92d9-93d3f5b260ba","Type":"ContainerStarted","Data":"57f2f70d5277f33fdf94f0227bf089c5a02ee9388da0a007fdf50d6dd96c3faa"} Nov 02 01:14:51.616981 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:51.616953 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/23bf37a9-9b32-47f9-a4e0-364a7273c02f-data-volume\") pod \"insights-runtime-extractor-gs62j\" (UID: \"23bf37a9-9b32-47f9-a4e0-364a7273c02f\") " pod="openshift-insights/insights-runtime-extractor-gs62j" Nov 02 01:14:51.617113 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:51.617013 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntxrh\" (UniqueName: \"kubernetes.io/projected/23bf37a9-9b32-47f9-a4e0-364a7273c02f-kube-api-access-ntxrh\") pod \"insights-runtime-extractor-gs62j\" (UID: \"23bf37a9-9b32-47f9-a4e0-364a7273c02f\") " pod="openshift-insights/insights-runtime-extractor-gs62j" Nov 02 01:14:51.617113 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:51.617070 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/23bf37a9-9b32-47f9-a4e0-364a7273c02f-crio-socket\") pod \"insights-runtime-extractor-gs62j\" (UID: \"23bf37a9-9b32-47f9-a4e0-364a7273c02f\") " pod="openshift-insights/insights-runtime-extractor-gs62j" Nov 02 01:14:51.617214 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:51.617113 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/23bf37a9-9b32-47f9-a4e0-364a7273c02f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gs62j\" (UID: \"23bf37a9-9b32-47f9-a4e0-364a7273c02f\") " pod="openshift-insights/insights-runtime-extractor-gs62j" Nov 02 01:14:51.617214 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:51.617137 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/23bf37a9-9b32-47f9-a4e0-364a7273c02f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gs62j\" (UID: \"23bf37a9-9b32-47f9-a4e0-364a7273c02f\") " pod="openshift-insights/insights-runtime-extractor-gs62j" Nov 02 01:14:51.617317 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:51.617292 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 01:14:51.617354 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:51.617349 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23bf37a9-9b32-47f9-a4e0-364a7273c02f-insights-runtime-extractor-tls podName:23bf37a9-9b32-47f9-a4e0-364a7273c02f nodeName:}" failed. No retries permitted until 2026-11-02 01:14:52.11733136 +0000 UTC m=+37.323624003 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/23bf37a9-9b32-47f9-a4e0-364a7273c02f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gs62j" (UID: "23bf37a9-9b32-47f9-a4e0-364a7273c02f") : secret "insights-runtime-extractor-tls" not found Nov 02 01:14:51.617818 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:51.617792 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/23bf37a9-9b32-47f9-a4e0-364a7273c02f-data-volume\") pod \"insights-runtime-extractor-gs62j\" (UID: \"23bf37a9-9b32-47f9-a4e0-364a7273c02f\") " pod="openshift-insights/insights-runtime-extractor-gs62j" Nov 02 01:14:51.618214 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:51.618185 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/23bf37a9-9b32-47f9-a4e0-364a7273c02f-crio-socket\") pod \"insights-runtime-extractor-gs62j\" (UID: \"23bf37a9-9b32-47f9-a4e0-364a7273c02f\") " pod="openshift-insights/insights-runtime-extractor-gs62j" Nov 02 01:14:51.618342 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:51.618213 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 01:14:51.618342 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:51.618271 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/23bf37a9-9b32-47f9-a4e0-364a7273c02f-kube-rbac-proxy-cm podName:23bf37a9-9b32-47f9-a4e0-364a7273c02f nodeName:}" failed. No retries permitted until 2026-11-02 01:14:52.118260098 +0000 UTC m=+37.324552766 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/23bf37a9-9b32-47f9-a4e0-364a7273c02f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gs62j" (UID: "23bf37a9-9b32-47f9-a4e0-364a7273c02f") : configmap "kube-rbac-proxy" not found Nov 02 01:14:51.629778 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:51.629711 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-pn4kl" podStartSLOduration=1.7115927690000001 podStartE2EDuration="6.629692776s" totalImagesPullingTime="4.918100007s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:14:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:14:51.628402353 +0000 UTC m=+36.834695010" watchObservedRunningTime="2026-11-02 01:14:51.629692776 +0000 UTC m=+36.835985439" Nov 02 01:14:51.651566 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:51.651496 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntxrh\" (UniqueName: \"kubernetes.io/projected/23bf37a9-9b32-47f9-a4e0-364a7273c02f-kube-api-access-ntxrh\") pod \"insights-runtime-extractor-gs62j\" (UID: \"23bf37a9-9b32-47f9-a4e0-364a7273c02f\") " pod="openshift-insights/insights-runtime-extractor-gs62j" Nov 02 01:14:51.717947 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:51.717916 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dg8x\" (UniqueName: \"kubernetes.io/projected/212ace3e-7830-4810-938d-22fd7f251b25-kube-api-access-7dg8x\") pod \"migrator-66df5d496b-ctstx\" (UID: \"212ace3e-7830-4810-938d-22fd7f251b25\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ctstx" Nov 02 01:14:51.818528 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:51.818491 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dg8x\" (UniqueName: \"kubernetes.io/projected/212ace3e-7830-4810-938d-22fd7f251b25-kube-api-access-7dg8x\") pod \"migrator-66df5d496b-ctstx\" (UID: \"212ace3e-7830-4810-938d-22fd7f251b25\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ctstx" Nov 02 01:14:51.834967 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:51.834938 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dg8x\" (UniqueName: \"kubernetes.io/projected/212ace3e-7830-4810-938d-22fd7f251b25-kube-api-access-7dg8x\") pod \"migrator-66df5d496b-ctstx\" (UID: \"212ace3e-7830-4810-938d-22fd7f251b25\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ctstx" Nov 02 01:14:51.899038 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:51.898997 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ctstx" Nov 02 01:14:52.120519 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:52.120472 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/23bf37a9-9b32-47f9-a4e0-364a7273c02f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gs62j\" (UID: \"23bf37a9-9b32-47f9-a4e0-364a7273c02f\") " pod="openshift-insights/insights-runtime-extractor-gs62j" Nov 02 01:14:52.120519 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:52.120520 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/23bf37a9-9b32-47f9-a4e0-364a7273c02f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gs62j\" (UID: \"23bf37a9-9b32-47f9-a4e0-364a7273c02f\") " pod="openshift-insights/insights-runtime-extractor-gs62j" Nov 02 01:14:52.120734 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:52.120598 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 01:14:52.120734 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:52.120629 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 01:14:52.120734 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:52.120665 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/23bf37a9-9b32-47f9-a4e0-364a7273c02f-kube-rbac-proxy-cm podName:23bf37a9-9b32-47f9-a4e0-364a7273c02f nodeName:}" failed. No retries permitted until 2026-11-02 01:14:53.120645525 +0000 UTC m=+38.326938188 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/23bf37a9-9b32-47f9-a4e0-364a7273c02f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gs62j" (UID: "23bf37a9-9b32-47f9-a4e0-364a7273c02f") : configmap "kube-rbac-proxy" not found Nov 02 01:14:52.120734 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:52.120679 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23bf37a9-9b32-47f9-a4e0-364a7273c02f-insights-runtime-extractor-tls podName:23bf37a9-9b32-47f9-a4e0-364a7273c02f nodeName:}" failed. No retries permitted until 2026-11-02 01:14:53.120673492 +0000 UTC m=+38.326966131 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/23bf37a9-9b32-47f9-a4e0-364a7273c02f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gs62j" (UID: "23bf37a9-9b32-47f9-a4e0-364a7273c02f") : secret "insights-runtime-extractor-tls" not found Nov 02 01:14:52.193753 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:52.193669 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-ctstx"] Nov 02 01:14:52.196843 ip-10-0-136-130 kubenswrapper[2578]: W1102 01:14:52.196811 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod212ace3e_7830_4810_938d_22fd7f251b25.slice/crio-fc108d6dd0a1a97b97982a28fea3da22b3dc06ac5e7b92c805628b335cb4b579 WatchSource:0}: Error finding container fc108d6dd0a1a97b97982a28fea3da22b3dc06ac5e7b92c805628b335cb4b579: Status 404 returned error can't find the container with id fc108d6dd0a1a97b97982a28fea3da22b3dc06ac5e7b92c805628b335cb4b579 Nov 02 01:14:52.612618 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:52.612570 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ctstx" event={"ID":"212ace3e-7830-4810-938d-22fd7f251b25","Type":"ContainerStarted","Data":"fc108d6dd0a1a97b97982a28fea3da22b3dc06ac5e7b92c805628b335cb4b579"} Nov 02 01:14:53.028681 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:53.028590 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/00d17dbb-cb72-4413-9559-4487c313158c-metrics-tls\") pod \"dns-default-xcm6b\" (UID: \"00d17dbb-cb72-4413-9559-4487c313158c\") " pod="openshift-dns/dns-default-xcm6b" Nov 02 01:14:53.028844 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:53.028758 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 01:14:53.028844 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:53.028828 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00d17dbb-cb72-4413-9559-4487c313158c-metrics-tls podName:00d17dbb-cb72-4413-9559-4487c313158c nodeName:}" failed. No retries permitted until 2026-11-02 01:15:01.028807893 +0000 UTC m=+46.235100556 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/00d17dbb-cb72-4413-9559-4487c313158c-metrics-tls") pod "dns-default-xcm6b" (UID: "00d17dbb-cb72-4413-9559-4487c313158c") : secret "dns-default-metrics-tls" not found Nov 02 01:14:53.130147 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:53.129529 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/23bf37a9-9b32-47f9-a4e0-364a7273c02f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gs62j\" (UID: \"23bf37a9-9b32-47f9-a4e0-364a7273c02f\") " pod="openshift-insights/insights-runtime-extractor-gs62j" Nov 02 01:14:53.130147 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:53.129575 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/23bf37a9-9b32-47f9-a4e0-364a7273c02f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gs62j\" (UID: \"23bf37a9-9b32-47f9-a4e0-364a7273c02f\") " pod="openshift-insights/insights-runtime-extractor-gs62j" Nov 02 01:14:53.130147 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:53.129733 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 01:14:53.130147 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:53.129789 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23bf37a9-9b32-47f9-a4e0-364a7273c02f-insights-runtime-extractor-tls podName:23bf37a9-9b32-47f9-a4e0-364a7273c02f nodeName:}" failed. No retries permitted until 2026-11-02 01:14:55.129775117 +0000 UTC m=+40.336067781 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/23bf37a9-9b32-47f9-a4e0-364a7273c02f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gs62j" (UID: "23bf37a9-9b32-47f9-a4e0-364a7273c02f") : secret "insights-runtime-extractor-tls" not found Nov 02 01:14:53.130147 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:53.130081 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 01:14:53.130147 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:53.130117 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/23bf37a9-9b32-47f9-a4e0-364a7273c02f-kube-rbac-proxy-cm podName:23bf37a9-9b32-47f9-a4e0-364a7273c02f nodeName:}" failed. No retries permitted until 2026-11-02 01:14:55.130106293 +0000 UTC m=+40.336398946 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/23bf37a9-9b32-47f9-a4e0-364a7273c02f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gs62j" (UID: "23bf37a9-9b32-47f9-a4e0-364a7273c02f") : configmap "kube-rbac-proxy" not found Nov 02 01:14:54.619558 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:54.619527 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fdptg" event={"ID":"3917d88c-88f5-4c5c-b1f7-e0195a61f88f","Type":"ContainerStarted","Data":"d99477d5d64115abec3fdaf7e9c49ffcb36e269e0282820cc993719505aad459"} Nov 02 01:14:54.620066 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:54.619654 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-fdptg" Nov 02 01:14:54.621117 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:54.621094 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ctstx" event={"ID":"212ace3e-7830-4810-938d-22fd7f251b25","Type":"ContainerStarted","Data":"bf32aec41573af2d85d33a16ad828e82b33998a19d5d450e14fa432adfbeede6"} Nov 02 01:14:54.641136 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:54.641079 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-fdptg" podStartSLOduration=35.577628825 podStartE2EDuration="39.641062489s" totalImagesPullingTime="4.063433664s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:14:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:14:54.640589337 +0000 UTC m=+39.846881999" watchObservedRunningTime="2026-11-02 01:14:54.641062489 +0000 UTC m=+39.847355151" Nov 02 01:14:54.658626 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:54.658576 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ctstx" podStartSLOduration=1.589310401 podStartE2EDuration="3.658561017s" totalImagesPullingTime="2.069250616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:14:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:14:54.6578495 +0000 UTC m=+39.864142194" watchObservedRunningTime="2026-11-02 01:14:54.658561017 +0000 UTC m=+39.864853724" Nov 02 01:14:55.145115 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:55.145085 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/23bf37a9-9b32-47f9-a4e0-364a7273c02f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gs62j\" (UID: \"23bf37a9-9b32-47f9-a4e0-364a7273c02f\") " pod="openshift-insights/insights-runtime-extractor-gs62j" Nov 02 01:14:55.145239 ip-10-0-136-130 kubenswrapper[2578]: I1102 01:14:55.145122 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/23bf37a9-9b32-47f9-a4e0-364a7273c02f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gs62j\" (UID: \"23bf37a9-9b32-47f9-a4e0-364a7273c02f\") " pod="openshift-insights/insights-runtime-extractor-gs62j" Nov 02 01:14:55.145239 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:55.145212 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 01:14:55.145350 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:55.145278 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/23bf37a9-9b32-47f9-a4e0-364a7273c02f-kube-rbac-proxy-cm podName:23bf37a9-9b32-47f9-a4e0-364a7273c02f nodeName:}" failed. No retries permitted until 2026-11-02 01:14:59.145263588 +0000 UTC m=+44.351556244 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/23bf37a9-9b32-47f9-a4e0-364a7273c02f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gs62j" (UID: "23bf37a9-9b32-47f9-a4e0-364a7273c02f") : configmap "kube-rbac-proxy" not found Nov 02 01:14:55.145350 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:55.145291 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 01:14:55.145457 ip-10-0-136-130 kubenswrapper[2578]: E1102 01:14:55.145343 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23bf37a9-9b32-47f9-a4e0-364a7273c02f-insights-runtime-extractor-tls podName:23bf37a9-9b32-47f9-a4e0-364a7273c02f nodeName:}" failed. No retries permitted until 2026-11-02 01:14:59.145326683 +0000 UTC m=+44.351619345 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/23bf37a9-9b32-47f9-a4e0-364a7273c02f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gs62j" (UID: "23bf37a9-9b32-47f9-a4e0-364a7273c02f") : secret "insights-runtime-extractor-tls" not found Nov 02 01:14:55.319553 ip-10-0-136-130 systemd[1]: Stopping Kubernetes Kubelet... Nov 02 01:14:55.333728 ip-10-0-136-130 systemd[1]: kubelet.service: Deactivated successfully. Nov 02 01:14:55.333893 ip-10-0-136-130 systemd[1]: Stopped Kubernetes Kubelet. Nov 02 01:14:55.333938 ip-10-0-136-130 systemd[1]: kubelet.service: Consumed 2.286s CPU time. Nov 02 01:14:55.347075 ip-10-0-136-130 systemd[1]: Starting Kubernetes Kubelet... Nov 02 01:14:55.445411 ip-10-0-136-130 kubenswrapper[4427]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 01:14:55.445411 ip-10-0-136-130 kubenswrapper[4427]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 01:14:55.445411 ip-10-0-136-130 kubenswrapper[4427]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 01:14:55.445411 ip-10-0-136-130 kubenswrapper[4427]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 01:14:55.447453 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447349 4427 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 01:14:55.447453 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447453 4427 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 01:14:55.447453 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447457 4427 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 01:14:55.447579 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447461 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 01:14:55.447579 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447465 4427 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 01:14:55.447579 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447468 4427 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 01:14:55.447579 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447471 4427 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 01:14:55.447579 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447474 4427 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 01:14:55.447579 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447476 4427 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 01:14:55.447579 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447479 4427 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 01:14:55.447579 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447482 4427 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 01:14:55.447579 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447484 4427 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 01:14:55.447579 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447487 4427 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 01:14:55.447579 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447490 4427 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 01:14:55.447579 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447493 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 01:14:55.447579 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447496 4427 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 01:14:55.447579 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447499 4427 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 01:14:55.447579 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447501 4427 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 01:14:55.447579 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447504 4427 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 01:14:55.447579 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447507 4427 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 01:14:55.447579 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447509 4427 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 01:14:55.447579 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447512 4427 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 01:14:55.447579 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447514 4427 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 01:14:55.448076 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447516 4427 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 01:14:55.448076 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447519 4427 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 01:14:55.448076 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447522 4427 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 01:14:55.448076 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447524 4427 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 01:14:55.448076 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447527 4427 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 01:14:55.448076 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447530 4427 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 01:14:55.448076 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447533 4427 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 01:14:55.448076 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447535 4427 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 01:14:55.448076 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447538 4427 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 01:14:55.448076 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447540 4427 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 01:14:55.448076 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447543 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 01:14:55.448076 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447545 4427 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 01:14:55.448076 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447548 4427 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 01:14:55.448076 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447550 4427 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 01:14:55.448076 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447553 4427 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 01:14:55.448076 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447556 4427 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 01:14:55.448076 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447558 4427 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 01:14:55.448076 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447561 4427 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 01:14:55.448076 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447564 4427 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 01:14:55.448076 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447567 4427 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 01:14:55.448591 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447569 4427 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 01:14:55.448591 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447571 4427 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 01:14:55.448591 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447574 4427 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 01:14:55.448591 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447576 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 01:14:55.448591 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447578 4427 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 01:14:55.448591 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447581 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 01:14:55.448591 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447584 4427 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 01:14:55.448591 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447586 4427 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 01:14:55.448591 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447589 4427 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 01:14:55.448591 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447592 4427 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 01:14:55.448591 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447595 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 01:14:55.448591 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447598 4427 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 01:14:55.448591 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447600 4427 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 01:14:55.448591 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447603 4427 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 01:14:55.448591 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447606 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 01:14:55.448591 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447608 4427 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 01:14:55.448591 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447611 4427 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 01:14:55.448591 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447614 4427 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 01:14:55.448591 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447616 4427 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 01:14:55.449099 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447618 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 01:14:55.449099 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447621 4427 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 01:14:55.449099 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447624 4427 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 01:14:55.449099 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447626 4427 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 01:14:55.449099 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447628 4427 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 01:14:55.449099 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447631 4427 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 01:14:55.449099 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447633 4427 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 01:14:55.449099 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447636 4427 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 01:14:55.449099 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447638 4427 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 01:14:55.449099 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447640 4427 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 01:14:55.449099 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447648 4427 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 01:14:55.449099 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447652 4427 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 01:14:55.449099 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447654 4427 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 01:14:55.449099 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447657 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 01:14:55.449099 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447659 4427 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 01:14:55.449099 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447662 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 01:14:55.449099 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447664 4427 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 01:14:55.449099 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447667 4427 feature_gate.go:386] unrecognized feature gate: Example Nov 02 01:14:55.449099 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447669 4427 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 01:14:55.449099 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447672 4427 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 01:14:55.449633 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447674 4427 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 01:14:55.449633 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447677 4427 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 01:14:55.449633 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447680 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 01:14:55.449633 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447685 4427 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 01:14:55.449633 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447688 4427 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 01:14:55.449633 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447692 4427 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 01:14:55.449633 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447696 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 01:14:55.449633 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447699 4427 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 01:14:55.449633 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447703 4427 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 01:14:55.449633 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447707 4427 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 01:14:55.449633 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447710 4427 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 01:14:55.449633 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447713 4427 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 01:14:55.449633 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447716 4427 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 01:14:55.449633 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447719 4427 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 01:14:55.449633 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447721 4427 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 01:14:55.449633 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447724 4427 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 01:14:55.449633 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447726 4427 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 01:14:55.449633 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447729 4427 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 01:14:55.450078 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447731 4427 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 01:14:55.450078 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447734 4427 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 01:14:55.450078 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447738 4427 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 01:14:55.450078 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447740 4427 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 01:14:55.450078 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447743 4427 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 01:14:55.450078 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.447746 4427 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 01:14:55.450078 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448489 4427 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 01:14:55.450078 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448496 4427 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 01:14:55.450078 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448499 4427 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 01:14:55.450078 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448503 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 01:14:55.450078 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448506 4427 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 01:14:55.450078 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448508 4427 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 01:14:55.450078 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448511 4427 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 01:14:55.450078 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448515 4427 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 01:14:55.450078 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448519 4427 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 01:14:55.450078 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448523 4427 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 01:14:55.450078 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448526 4427 feature_gate.go:386] unrecognized feature gate: Example Nov 02 01:14:55.450078 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448531 4427 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 01:14:55.450078 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448533 4427 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 01:14:55.450566 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448536 4427 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 01:14:55.450566 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448538 4427 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 01:14:55.450566 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448541 4427 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 01:14:55.450566 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448543 4427 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 01:14:55.450566 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448546 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 01:14:55.450566 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448549 4427 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 01:14:55.450566 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448551 4427 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 01:14:55.450566 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448554 4427 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 01:14:55.450566 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448556 4427 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 01:14:55.450566 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448558 4427 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 01:14:55.450566 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448561 4427 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 01:14:55.450566 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448564 4427 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 01:14:55.450566 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448568 4427 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 01:14:55.450566 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448571 4427 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 01:14:55.450566 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448574 4427 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 01:14:55.450566 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448576 4427 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 01:14:55.450566 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448579 4427 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 01:14:55.450566 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448581 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 01:14:55.450566 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448584 4427 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 01:14:55.451047 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448587 4427 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 01:14:55.451047 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448590 4427 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 01:14:55.451047 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448592 4427 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 01:14:55.451047 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448595 4427 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 01:14:55.451047 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448597 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 01:14:55.451047 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448600 4427 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 01:14:55.451047 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448603 4427 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 01:14:55.451047 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448605 4427 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 01:14:55.451047 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448608 4427 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 01:14:55.451047 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448611 4427 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 01:14:55.451047 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448614 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 01:14:55.451047 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448618 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 01:14:55.451047 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448620 4427 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 01:14:55.451047 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448623 4427 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 01:14:55.451047 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448625 4427 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 01:14:55.451047 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448628 4427 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 01:14:55.451047 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448630 4427 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 01:14:55.451047 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448633 4427 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 01:14:55.451047 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448635 4427 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 01:14:55.451047 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448638 4427 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 01:14:55.451582 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448640 4427 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 01:14:55.451582 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448643 4427 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 01:14:55.451582 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448645 4427 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 01:14:55.451582 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448648 4427 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 01:14:55.451582 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448651 4427 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 01:14:55.451582 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448653 4427 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 01:14:55.451582 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448656 4427 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 01:14:55.451582 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448658 4427 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 01:14:55.451582 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448660 4427 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 01:14:55.451582 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448663 4427 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 01:14:55.451582 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448665 4427 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 01:14:55.451582 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448668 4427 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 01:14:55.451582 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448670 4427 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 01:14:55.451582 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448673 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 01:14:55.451582 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448676 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 01:14:55.451582 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448679 4427 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 01:14:55.451582 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448681 4427 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 01:14:55.451582 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448683 4427 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 01:14:55.451582 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448686 4427 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 01:14:55.452049 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448688 4427 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 01:14:55.452049 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448691 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 01:14:55.452049 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448693 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 01:14:55.452049 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448696 4427 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 01:14:55.452049 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448700 4427 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 01:14:55.452049 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448703 4427 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 01:14:55.452049 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448706 4427 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 01:14:55.452049 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448708 4427 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 01:14:55.452049 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448710 4427 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 01:14:55.452049 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448713 4427 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 01:14:55.452049 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448715 4427 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 01:14:55.452049 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448718 4427 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 01:14:55.452049 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448720 4427 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 01:14:55.452049 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448723 4427 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 01:14:55.452049 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448725 4427 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 01:14:55.452049 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448728 4427 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 01:14:55.452049 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448731 4427 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 01:14:55.452049 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448733 4427 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 01:14:55.452049 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448736 4427 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 01:14:55.452049 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448738 4427 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 01:14:55.452562 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448741 4427 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 01:14:55.452562 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448743 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 01:14:55.452562 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448745 4427 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 01:14:55.452562 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448748 4427 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 01:14:55.452562 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448750 4427 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 01:14:55.452562 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448754 4427 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 01:14:55.452562 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448756 4427 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 01:14:55.452562 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448759 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 01:14:55.452562 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448762 4427 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 01:14:55.452562 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448764 4427 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 01:14:55.452562 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448766 4427 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 01:14:55.452562 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448769 4427 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 01:14:55.452562 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448771 4427 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 01:14:55.452562 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448774 4427 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 01:14:55.452562 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.448776 4427 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 01:14:55.452562 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449266 4427 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 01:14:55.452562 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449273 4427 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 01:14:55.452562 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449276 4427 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 01:14:55.452562 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449278 4427 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 01:14:55.452562 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449281 4427 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 01:14:55.453065 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449283 4427 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 01:14:55.453065 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449286 4427 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 01:14:55.453065 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449288 4427 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 01:14:55.453065 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449291 4427 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 01:14:55.453065 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449295 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 01:14:55.453065 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449297 4427 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 01:14:55.453065 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449300 4427 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 01:14:55.453065 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449302 4427 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 01:14:55.453065 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449305 4427 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 01:14:55.453065 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449307 4427 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 01:14:55.453065 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449310 4427 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 01:14:55.453065 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449313 4427 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 01:14:55.453065 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449315 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 01:14:55.453065 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449318 4427 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 01:14:55.453065 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449320 4427 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 01:14:55.453065 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449323 4427 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 01:14:55.453065 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449325 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 01:14:55.453065 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449328 4427 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 01:14:55.453065 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449330 4427 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 01:14:55.453065 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449334 4427 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 01:14:55.453870 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449337 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 01:14:55.453870 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449340 4427 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 01:14:55.453870 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449342 4427 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 01:14:55.453870 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449344 4427 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 01:14:55.453870 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449347 4427 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 01:14:55.453870 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449349 4427 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 01:14:55.453870 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449352 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 01:14:55.453870 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449355 4427 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 01:14:55.453870 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449359 4427 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 01:14:55.453870 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449362 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 01:14:55.453870 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449364 4427 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 01:14:55.453870 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449367 4427 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 01:14:55.453870 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449385 4427 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 01:14:55.453870 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449388 4427 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 01:14:55.453870 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449390 4427 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 01:14:55.453870 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449393 4427 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 01:14:55.453870 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449395 4427 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 01:14:55.453870 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449398 4427 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 01:14:55.453870 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449401 4427 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 01:14:55.453870 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449403 4427 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 01:14:55.454750 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449406 4427 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 01:14:55.454750 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449409 4427 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 01:14:55.454750 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449411 4427 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 01:14:55.454750 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449414 4427 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 01:14:55.454750 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449416 4427 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 01:14:55.454750 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449419 4427 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 01:14:55.454750 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449421 4427 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 01:14:55.454750 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449424 4427 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 01:14:55.454750 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449426 4427 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 01:14:55.454750 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449429 4427 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 01:14:55.454750 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449431 4427 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 01:14:55.454750 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449434 4427 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 01:14:55.454750 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449437 4427 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 01:14:55.454750 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449440 4427 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 01:14:55.454750 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449443 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 01:14:55.454750 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449445 4427 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 01:14:55.454750 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449448 4427 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 01:14:55.454750 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449450 4427 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 01:14:55.454750 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449453 4427 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 01:14:55.454750 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449456 4427 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 01:14:55.455911 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449459 4427 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 01:14:55.455911 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449462 4427 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 01:14:55.455911 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449464 4427 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 01:14:55.455911 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449467 4427 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 01:14:55.455911 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449469 4427 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 01:14:55.455911 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449472 4427 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 01:14:55.455911 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449474 4427 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 01:14:55.455911 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449477 4427 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 01:14:55.455911 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449479 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 01:14:55.455911 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449483 4427 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 01:14:55.455911 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449487 4427 feature_gate.go:386] unrecognized feature gate: Example Nov 02 01:14:55.455911 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449490 4427 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 01:14:55.455911 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449494 4427 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 01:14:55.455911 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449497 4427 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 01:14:55.455911 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449500 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 01:14:55.455911 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449503 4427 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 01:14:55.455911 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449506 4427 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 01:14:55.455911 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449508 4427 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 01:14:55.455911 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449510 4427 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 01:14:55.456763 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449513 4427 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 01:14:55.456763 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449515 4427 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 01:14:55.456763 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449518 4427 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 01:14:55.456763 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449520 4427 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 01:14:55.456763 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449523 4427 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 01:14:55.456763 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449526 4427 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 01:14:55.456763 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449530 4427 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 01:14:55.456763 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449533 4427 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 01:14:55.456763 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449536 4427 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 01:14:55.456763 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449538 4427 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 01:14:55.456763 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449541 4427 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 01:14:55.456763 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449543 4427 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 01:14:55.456763 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449546 4427 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 01:14:55.456763 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449550 4427 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 01:14:55.456763 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449552 4427 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 01:14:55.456763 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449555 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 01:14:55.456763 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449557 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 01:14:55.456763 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449560 4427 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 01:14:55.456763 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449562 4427 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 01:14:55.457621 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449565 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 01:14:55.457621 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449567 4427 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 01:14:55.457621 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.449571 4427 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 01:14:55.457621 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449662 4427 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 01:14:55.457621 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449678 4427 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 01:14:55.457621 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449687 4427 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 01:14:55.457621 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449694 4427 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 01:14:55.457621 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449701 4427 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 01:14:55.457621 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449705 4427 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 01:14:55.457621 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449710 4427 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 01:14:55.457621 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449718 4427 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 01:14:55.457621 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449721 4427 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 01:14:55.457621 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449725 4427 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 01:14:55.457621 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449728 4427 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 01:14:55.457621 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449731 4427 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 01:14:55.457621 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449734 4427 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 01:14:55.457621 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449737 4427 flags.go:64] FLAG: --cgroup-root="" Nov 02 01:14:55.457621 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449740 4427 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 01:14:55.457621 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449743 4427 flags.go:64] FLAG: --client-ca-file="" Nov 02 01:14:55.457621 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449746 4427 flags.go:64] FLAG: --cloud-provider="external" Nov 02 01:14:55.457621 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449750 4427 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 01:14:55.457621 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449754 4427 flags.go:64] FLAG: --cluster-domain="" Nov 02 01:14:55.457621 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449757 4427 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 01:14:55.457621 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449760 4427 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 01:14:55.458550 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449763 4427 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 01:14:55.458550 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449767 4427 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 01:14:55.458550 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449771 4427 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 01:14:55.458550 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449774 4427 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 01:14:55.458550 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449777 4427 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 01:14:55.458550 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449780 4427 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 01:14:55.458550 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449783 4427 flags.go:64] FLAG: --contention-profiling="false" Nov 02 01:14:55.458550 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449786 4427 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 01:14:55.458550 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449789 4427 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 01:14:55.458550 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449792 4427 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 01:14:55.458550 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449795 4427 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 01:14:55.458550 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449799 4427 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 01:14:55.458550 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449802 4427 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 01:14:55.458550 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449805 4427 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 01:14:55.458550 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449808 4427 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 01:14:55.458550 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449813 4427 flags.go:64] FLAG: --enable-server="true" Nov 02 01:14:55.458550 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449816 4427 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 01:14:55.458550 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449821 4427 flags.go:64] FLAG: --event-burst="100" Nov 02 01:14:55.458550 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449824 4427 flags.go:64] FLAG: --event-qps="50" Nov 02 01:14:55.458550 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449827 4427 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 01:14:55.458550 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449830 4427 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 01:14:55.458550 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449832 4427 flags.go:64] FLAG: --eviction-hard="" Nov 02 01:14:55.458550 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449837 4427 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 01:14:55.458550 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449840 4427 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 01:14:55.459154 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449842 4427 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 01:14:55.459154 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449845 4427 flags.go:64] FLAG: --eviction-soft="" Nov 02 01:14:55.459154 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449848 4427 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 01:14:55.459154 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449851 4427 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 01:14:55.459154 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449854 4427 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 01:14:55.459154 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449857 4427 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 01:14:55.459154 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449860 4427 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 01:14:55.459154 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449864 4427 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 01:14:55.459154 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449867 4427 flags.go:64] FLAG: --feature-gates="" Nov 02 01:14:55.459154 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449871 4427 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 01:14:55.459154 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449874 4427 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 01:14:55.459154 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449877 4427 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 01:14:55.459154 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449880 4427 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 01:14:55.459154 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449883 4427 flags.go:64] FLAG: --healthz-port="10248" Nov 02 01:14:55.459154 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449886 4427 flags.go:64] FLAG: --help="false" Nov 02 01:14:55.459154 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449889 4427 flags.go:64] FLAG: --hostname-override="ip-10-0-136-130.us-east-2.compute.internal" Nov 02 01:14:55.459154 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449892 4427 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 01:14:55.459154 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449895 4427 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 01:14:55.459154 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449898 4427 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 01:14:55.459154 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449901 4427 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 01:14:55.459154 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449905 4427 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 01:14:55.459154 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449907 4427 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 01:14:55.459154 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449910 4427 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 01:14:55.459742 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449914 4427 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 01:14:55.459742 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449917 4427 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 01:14:55.459742 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449920 4427 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 01:14:55.459742 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449923 4427 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 01:14:55.459742 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449926 4427 flags.go:64] FLAG: --kube-reserved="" Nov 02 01:14:55.459742 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449929 4427 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 01:14:55.459742 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449932 4427 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 01:14:55.459742 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449935 4427 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 01:14:55.459742 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449938 4427 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 01:14:55.459742 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449940 4427 flags.go:64] FLAG: --lock-file="" Nov 02 01:14:55.459742 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449943 4427 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 01:14:55.459742 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449946 4427 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 01:14:55.459742 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449949 4427 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 01:14:55.459742 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449954 4427 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 01:14:55.459742 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449957 4427 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 01:14:55.459742 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449961 4427 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 01:14:55.459742 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449964 4427 flags.go:64] FLAG: --logging-format="text" Nov 02 01:14:55.459742 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449967 4427 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 01:14:55.459742 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449970 4427 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 01:14:55.459742 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449973 4427 flags.go:64] FLAG: --manifest-url="" Nov 02 01:14:55.459742 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449976 4427 flags.go:64] FLAG: --manifest-url-header="" Nov 02 01:14:55.459742 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449980 4427 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 01:14:55.459742 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449983 4427 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 01:14:55.459742 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449987 4427 flags.go:64] FLAG: --max-pods="110" Nov 02 01:14:55.459742 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449990 4427 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 01:14:55.460608 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449993 4427 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 01:14:55.460608 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449996 4427 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 01:14:55.460608 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.449999 4427 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 01:14:55.460608 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.450002 4427 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 01:14:55.460608 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.450004 4427 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 01:14:55.460608 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.450007 4427 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 01:14:55.460608 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.450014 4427 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 01:14:55.460608 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.450021 4427 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 01:14:55.460608 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.450024 4427 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 01:14:55.460608 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.450027 4427 flags.go:64] FLAG: --pod-cidr="" Nov 02 01:14:55.460608 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.450029 4427 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 01:14:55.460608 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.450032 4427 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 01:14:55.460608 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.450035 4427 flags.go:64] FLAG: --pods-per-core="0" Nov 02 01:14:55.460608 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.450038 4427 flags.go:64] FLAG: --port="10250" Nov 02 01:14:55.460608 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.450041 4427 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 01:14:55.460608 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.450044 4427 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-057ef1ed1b55324a5" Nov 02 01:14:55.460608 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.450047 4427 flags.go:64] FLAG: --qos-reserved="" Nov 02 01:14:55.460608 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.450050 4427 flags.go:64] FLAG: --read-only-port="10255" Nov 02 01:14:55.460608 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.450052 4427 flags.go:64] FLAG: --register-node="true" Nov 02 01:14:55.460608 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.450055 4427 flags.go:64] FLAG: --register-with-taints="" Nov 02 01:14:55.460608 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.450059 4427 flags.go:64] FLAG: --registry-burst="10" Nov 02 01:14:55.460608 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.450062 4427 flags.go:64] FLAG: --registry-qps="5" Nov 02 01:14:55.460608 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.450064 4427 flags.go:64] FLAG: --reserved-cpus="" Nov 02 01:14:55.460608 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.450068 4427 flags.go:64] FLAG: --reserved-memory="" Nov 02 01:14:55.460608 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.450072 4427 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 01:14:55.460608 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.450075 4427 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 01:14:55.461248 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.450078 4427 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 01:14:55.461248 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.450082 4427 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 01:14:55.461248 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.450085 4427 flags.go:64] FLAG: --runonce="false" Nov 02 01:14:55.461248 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.450088 4427 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 01:14:55.461248 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.450091 4427 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 01:14:55.461248 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.450094 4427 flags.go:64] FLAG: --seccomp-default="false" Nov 02 01:14:55.461248 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.450097 4427 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 01:14:55.461248 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.450099 4427 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 01:14:55.461248 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.450102 4427 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 01:14:55.461248 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.450105 4427 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 01:14:55.461248 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.450108 4427 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 01:14:55.461248 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.450111 4427 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 01:14:55.461248 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.450114 4427 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 01:14:55.461248 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.450119 4427 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 01:14:55.461248 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.450121 4427 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 01:14:55.461248 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.450126 4427 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 01:14:55.461248 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.450129 4427 flags.go:64] FLAG: --system-cgroups="" Nov 02 01:14:55.461248 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.450132 4427 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 01:14:55.461248 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.450136 4427 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 01:14:55.461248 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.450139 4427 flags.go:64] FLAG: --tls-cert-file="" Nov 02 01:14:55.461248 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.450142 4427 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 01:14:55.461248 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.450146 4427 flags.go:64] FLAG: --tls-min-version="" Nov 02 01:14:55.461248 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.450149 4427 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 01:14:55.461248 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.450151 4427 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 01:14:55.461248 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.450154 4427 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 01:14:55.462176 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.450157 4427 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 01:14:55.462176 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.450160 4427 flags.go:64] FLAG: --v="2" Nov 02 01:14:55.462176 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.450164 4427 flags.go:64] FLAG: --version="false" Nov 02 01:14:55.462176 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.450168 4427 flags.go:64] FLAG: --vmodule="" Nov 02 01:14:55.462176 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.450172 4427 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 01:14:55.462176 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.450176 4427 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 01:14:55.462176 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450306 4427 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 01:14:55.462176 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450310 4427 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 01:14:55.462176 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450313 4427 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 01:14:55.462176 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450316 4427 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 01:14:55.462176 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450319 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 01:14:55.462176 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450322 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 01:14:55.462176 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450324 4427 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 01:14:55.462176 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450327 4427 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 01:14:55.462176 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450329 4427 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 01:14:55.462176 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450333 4427 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 01:14:55.462176 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450337 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 01:14:55.462176 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450339 4427 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 01:14:55.462176 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450342 4427 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 01:14:55.462176 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450344 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 01:14:55.462980 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450349 4427 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 01:14:55.462980 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450351 4427 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 01:14:55.462980 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450355 4427 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 01:14:55.462980 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450358 4427 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 01:14:55.462980 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450360 4427 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 01:14:55.462980 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450363 4427 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 01:14:55.462980 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450365 4427 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 01:14:55.462980 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450368 4427 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 01:14:55.462980 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450385 4427 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 01:14:55.462980 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450387 4427 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 01:14:55.462980 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450390 4427 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 01:14:55.462980 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450393 4427 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 01:14:55.462980 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450395 4427 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 01:14:55.462980 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450399 4427 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 01:14:55.462980 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450401 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 01:14:55.462980 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450406 4427 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 01:14:55.462980 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450410 4427 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 01:14:55.462980 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450413 4427 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 01:14:55.462980 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450417 4427 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 01:14:55.462980 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450419 4427 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 01:14:55.463627 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450422 4427 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 01:14:55.463627 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450425 4427 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 01:14:55.463627 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450427 4427 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 01:14:55.463627 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450431 4427 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 01:14:55.463627 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450433 4427 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 01:14:55.463627 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450436 4427 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 01:14:55.463627 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450438 4427 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 01:14:55.463627 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450441 4427 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 01:14:55.463627 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450443 4427 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 01:14:55.463627 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450446 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 01:14:55.463627 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450448 4427 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 01:14:55.463627 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450451 4427 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 01:14:55.463627 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450455 4427 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 01:14:55.463627 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450457 4427 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 01:14:55.463627 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450461 4427 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 01:14:55.463627 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450464 4427 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 01:14:55.463627 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450466 4427 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 01:14:55.463627 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450469 4427 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 01:14:55.463627 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450471 4427 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 01:14:55.463627 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450474 4427 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 01:14:55.464180 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450477 4427 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 01:14:55.464180 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450479 4427 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 01:14:55.464180 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450482 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 01:14:55.464180 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450484 4427 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 01:14:55.464180 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450487 4427 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 01:14:55.464180 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450489 4427 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 01:14:55.464180 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450492 4427 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 01:14:55.464180 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450495 4427 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 01:14:55.464180 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450497 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 01:14:55.464180 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450500 4427 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 01:14:55.464180 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450502 4427 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 01:14:55.464180 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450505 4427 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 01:14:55.464180 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450508 4427 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 01:14:55.464180 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450510 4427 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 01:14:55.464180 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450513 4427 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 01:14:55.464180 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450515 4427 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 01:14:55.464180 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450517 4427 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 01:14:55.464180 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450520 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 01:14:55.464180 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450522 4427 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 01:14:55.464180 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450525 4427 feature_gate.go:386] unrecognized feature gate: Example Nov 02 01:14:55.464749 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450527 4427 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 01:14:55.464749 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450530 4427 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 01:14:55.464749 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450532 4427 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 01:14:55.464749 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450535 4427 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 01:14:55.464749 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450539 4427 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 01:14:55.464749 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450542 4427 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 01:14:55.464749 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450545 4427 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 01:14:55.464749 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450548 4427 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 01:14:55.464749 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450550 4427 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 01:14:55.464749 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450553 4427 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 01:14:55.464749 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450556 4427 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 01:14:55.464749 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450558 4427 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 01:14:55.464749 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450561 4427 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 01:14:55.464749 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450563 4427 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 01:14:55.464749 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450566 4427 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 01:14:55.464749 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450568 4427 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 01:14:55.464749 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450571 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 01:14:55.464749 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450573 4427 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 01:14:55.464749 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450576 4427 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 01:14:55.465268 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450579 4427 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 01:14:55.465268 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450581 4427 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 01:14:55.465268 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450584 4427 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 01:14:55.465268 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450586 4427 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 01:14:55.465268 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450589 4427 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 01:14:55.465268 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450592 4427 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 01:14:55.465268 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450594 4427 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 01:14:55.465268 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450597 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 01:14:55.465268 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450599 4427 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 01:14:55.465268 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450601 4427 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 01:14:55.465268 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450604 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 01:14:55.465268 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450607 4427 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 01:14:55.465268 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450609 4427 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 01:14:55.465268 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.450635 4427 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 01:14:55.465268 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450669 4427 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 01:14:55.465268 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450673 4427 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 01:14:55.465713 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450676 4427 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 01:14:55.465713 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450680 4427 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 01:14:55.465713 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450682 4427 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 01:14:55.465713 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450685 4427 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 01:14:55.465713 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450688 4427 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 01:14:55.465713 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450691 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 01:14:55.465713 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450693 4427 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 01:14:55.465713 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450696 4427 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 01:14:55.465713 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450698 4427 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 01:14:55.465713 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450701 4427 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 01:14:55.465713 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450703 4427 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 01:14:55.465713 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450706 4427 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 01:14:55.465713 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450708 4427 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 01:14:55.465713 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450711 4427 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 01:14:55.465713 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450714 4427 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 01:14:55.465713 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450716 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 01:14:55.465713 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450718 4427 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 01:14:55.465713 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450721 4427 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 01:14:55.465713 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450724 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 01:14:55.465713 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450726 4427 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 01:14:55.466208 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450729 4427 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 01:14:55.466208 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450731 4427 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 01:14:55.466208 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450742 4427 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 01:14:55.466208 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450745 4427 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 01:14:55.466208 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450747 4427 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 01:14:55.466208 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450750 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 01:14:55.466208 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450752 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 01:14:55.466208 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450755 4427 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 01:14:55.466208 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450757 4427 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 01:14:55.466208 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450760 4427 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 01:14:55.466208 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450764 4427 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 01:14:55.466208 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450767 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 01:14:55.466208 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450770 4427 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 01:14:55.466208 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450774 4427 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 01:14:55.466208 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450777 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 01:14:55.466208 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450779 4427 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 01:14:55.466208 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450783 4427 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 01:14:55.466208 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450785 4427 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 01:14:55.466208 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450788 4427 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 01:14:55.466688 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450790 4427 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 01:14:55.466688 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450793 4427 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 01:14:55.466688 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450796 4427 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 01:14:55.466688 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450798 4427 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 01:14:55.466688 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450801 4427 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 01:14:55.466688 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450803 4427 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 01:14:55.466688 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450805 4427 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 01:14:55.466688 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450808 4427 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 01:14:55.466688 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450810 4427 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 01:14:55.466688 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450813 4427 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 01:14:55.466688 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450815 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 01:14:55.466688 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450819 4427 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 01:14:55.466688 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450823 4427 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 01:14:55.466688 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450827 4427 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 01:14:55.466688 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450830 4427 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 01:14:55.466688 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450833 4427 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 01:14:55.466688 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450835 4427 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 01:14:55.466688 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450838 4427 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 01:14:55.466688 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450840 4427 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 01:14:55.466688 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450843 4427 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 01:14:55.467176 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450845 4427 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 01:14:55.467176 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450848 4427 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 01:14:55.467176 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450850 4427 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 01:14:55.467176 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450853 4427 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 01:14:55.467176 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450855 4427 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 01:14:55.467176 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450858 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 01:14:55.467176 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450861 4427 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 01:14:55.467176 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450864 4427 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 01:14:55.467176 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450867 4427 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 01:14:55.467176 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450871 4427 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 01:14:55.467176 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450874 4427 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 01:14:55.467176 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450876 4427 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 01:14:55.467176 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450879 4427 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 01:14:55.467176 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450881 4427 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 01:14:55.467176 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450884 4427 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 01:14:55.467176 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450887 4427 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 01:14:55.467176 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450889 4427 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 01:14:55.467176 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450892 4427 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 01:14:55.467176 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450894 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 01:14:55.467176 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450897 4427 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 01:14:55.467799 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450900 4427 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 01:14:55.467799 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450902 4427 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 01:14:55.467799 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450904 4427 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 01:14:55.467799 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450907 4427 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 01:14:55.467799 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450909 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 01:14:55.467799 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450912 4427 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 01:14:55.467799 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450914 4427 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 01:14:55.467799 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450924 4427 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 01:14:55.467799 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450928 4427 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 01:14:55.467799 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450931 4427 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 01:14:55.467799 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450934 4427 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 01:14:55.467799 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450937 4427 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 01:14:55.467799 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450939 4427 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 01:14:55.467799 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450942 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 01:14:55.467799 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450944 4427 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 01:14:55.467799 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450947 4427 feature_gate.go:386] unrecognized feature gate: Example Nov 02 01:14:55.467799 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450949 4427 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 01:14:55.467799 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450952 4427 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 01:14:55.467799 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450956 4427 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 01:14:55.467799 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450958 4427 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 01:14:55.468284 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450961 4427 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 01:14:55.468284 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450967 4427 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 01:14:55.468284 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450969 4427 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 01:14:55.468284 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450972 4427 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 01:14:55.468284 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.450974 4427 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 01:14:55.468284 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.455763 4427 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 01:14:55.468284 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.455783 4427 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 01:14:55.468284 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.455807 4427 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 01:14:55.468284 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.455815 4427 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 01:14:55.468284 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.455899 4427 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 01:14:55.468284 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.455907 4427 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 01:14:55.468284 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.455913 4427 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 01:14:55.468284 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.455918 4427 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 01:14:55.468284 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.455922 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 01:14:55.468284 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.455927 4427 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 01:14:55.468284 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.455931 4427 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 01:14:55.468284 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.455935 4427 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 01:14:55.468284 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.455939 4427 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 01:14:55.468284 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.455944 4427 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 01:14:55.468770 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.455948 4427 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 01:14:55.468770 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.455953 4427 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 01:14:55.468770 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.455957 4427 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 01:14:55.468770 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.455962 4427 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 01:14:55.468770 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.455966 4427 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 01:14:55.468770 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.455970 4427 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 01:14:55.468770 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.455974 4427 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 01:14:55.468770 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.455978 4427 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 01:14:55.468770 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.455982 4427 feature_gate.go:386] unrecognized feature gate: Example Nov 02 01:14:55.468770 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.455987 4427 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 01:14:55.468770 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.455991 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 01:14:55.468770 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.455995 4427 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 01:14:55.468770 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456000 4427 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 01:14:55.468770 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456005 4427 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 01:14:55.468770 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456009 4427 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 01:14:55.468770 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456014 4427 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 01:14:55.468770 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456018 4427 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 01:14:55.468770 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456023 4427 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 01:14:55.468770 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456028 4427 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 01:14:55.468770 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456034 4427 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 01:14:55.469255 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456038 4427 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 01:14:55.469255 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456043 4427 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 01:14:55.469255 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456047 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 01:14:55.469255 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456051 4427 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 01:14:55.469255 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456055 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 01:14:55.469255 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456059 4427 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 01:14:55.469255 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456063 4427 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 01:14:55.469255 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456068 4427 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 01:14:55.469255 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456072 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 01:14:55.469255 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456076 4427 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 01:14:55.469255 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456080 4427 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 01:14:55.469255 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456084 4427 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 01:14:55.469255 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456088 4427 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 01:14:55.469255 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456093 4427 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 01:14:55.469255 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456097 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 01:14:55.469255 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456101 4427 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 01:14:55.469255 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456105 4427 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 01:14:55.469255 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456109 4427 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 01:14:55.469255 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456114 4427 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 01:14:55.469255 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456118 4427 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 01:14:55.469804 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456122 4427 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 01:14:55.469804 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456126 4427 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 01:14:55.469804 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456130 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 01:14:55.469804 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456134 4427 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 01:14:55.469804 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456139 4427 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 01:14:55.469804 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456144 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 01:14:55.469804 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456148 4427 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 01:14:55.469804 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456152 4427 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 01:14:55.469804 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456173 4427 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 01:14:55.469804 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456178 4427 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 01:14:55.469804 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456182 4427 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 01:14:55.469804 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456186 4427 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 01:14:55.469804 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456191 4427 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 01:14:55.469804 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456195 4427 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 01:14:55.469804 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456200 4427 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 01:14:55.469804 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456204 4427 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 01:14:55.469804 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456208 4427 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 01:14:55.469804 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456213 4427 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 01:14:55.469804 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456217 4427 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 01:14:55.469804 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456221 4427 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 01:14:55.470283 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456226 4427 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 01:14:55.470283 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456230 4427 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 01:14:55.470283 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456235 4427 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 01:14:55.470283 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456240 4427 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 01:14:55.470283 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456244 4427 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 01:14:55.470283 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456248 4427 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 01:14:55.470283 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456254 4427 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 01:14:55.470283 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456261 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 01:14:55.470283 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456266 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 01:14:55.470283 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456271 4427 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 01:14:55.470283 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456275 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 01:14:55.470283 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456280 4427 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 01:14:55.470283 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456284 4427 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 01:14:55.470283 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456288 4427 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 01:14:55.470283 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456293 4427 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 01:14:55.470283 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456297 4427 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 01:14:55.470283 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456301 4427 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 01:14:55.470283 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456306 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 01:14:55.470283 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456310 4427 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 01:14:55.470764 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456317 4427 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 01:14:55.470764 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456323 4427 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 01:14:55.470764 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456328 4427 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 01:14:55.470764 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456333 4427 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 01:14:55.470764 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456337 4427 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 01:14:55.470764 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456341 4427 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 01:14:55.470764 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456346 4427 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 01:14:55.470764 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456350 4427 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 01:14:55.470764 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456355 4427 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 01:14:55.470764 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456359 4427 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 01:14:55.470764 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456363 4427 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 01:14:55.470764 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456367 4427 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 01:14:55.470764 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456389 4427 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 01:14:55.470764 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456394 4427 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 01:14:55.470764 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456398 4427 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 01:14:55.470764 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456402 4427 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 01:14:55.470764 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456406 4427 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 01:14:55.471185 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.456448 4427 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 01:14:55.471185 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456708 4427 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 01:14:55.471185 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456718 4427 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 01:14:55.471185 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456723 4427 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 01:14:55.471185 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456727 4427 feature_gate.go:386] unrecognized feature gate: Example Nov 02 01:14:55.471185 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456732 4427 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 01:14:55.471185 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456736 4427 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 01:14:55.471185 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456741 4427 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 01:14:55.471185 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456745 4427 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 01:14:55.471185 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456749 4427 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 01:14:55.471185 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456753 4427 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 01:14:55.471185 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456757 4427 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 01:14:55.471185 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456762 4427 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 01:14:55.471185 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456766 4427 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 01:14:55.471185 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456771 4427 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 01:14:55.471185 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456776 4427 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 01:14:55.471614 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456780 4427 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 01:14:55.471614 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456783 4427 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 01:14:55.471614 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456787 4427 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 01:14:55.471614 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456791 4427 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 01:14:55.471614 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456795 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 01:14:55.471614 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456800 4427 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 01:14:55.471614 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456805 4427 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 01:14:55.471614 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456810 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 01:14:55.471614 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456814 4427 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 01:14:55.471614 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456819 4427 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 01:14:55.471614 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456823 4427 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 01:14:55.471614 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456827 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 01:14:55.471614 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456831 4427 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 01:14:55.471614 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456835 4427 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 01:14:55.471614 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456842 4427 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 01:14:55.471614 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456849 4427 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 01:14:55.471614 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456854 4427 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 01:14:55.471614 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456858 4427 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 01:14:55.471614 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456862 4427 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 01:14:55.472073 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456867 4427 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 01:14:55.472073 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456873 4427 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 01:14:55.472073 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456878 4427 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 01:14:55.472073 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456883 4427 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 01:14:55.472073 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456887 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 01:14:55.472073 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456892 4427 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 01:14:55.472073 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456896 4427 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 01:14:55.472073 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456900 4427 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 01:14:55.472073 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456904 4427 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 01:14:55.472073 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456909 4427 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 01:14:55.472073 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456913 4427 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 01:14:55.472073 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456918 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 01:14:55.472073 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456922 4427 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 01:14:55.472073 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456926 4427 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 01:14:55.472073 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456931 4427 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 01:14:55.472073 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456935 4427 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 01:14:55.472073 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456939 4427 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 01:14:55.472073 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456944 4427 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 01:14:55.472073 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456948 4427 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 01:14:55.472562 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.456952 4427 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 01:14:55.472562 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457014 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 01:14:55.472562 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457126 4427 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 01:14:55.472562 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457132 4427 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 01:14:55.472562 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457136 4427 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 01:14:55.472562 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457142 4427 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 01:14:55.472562 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457147 4427 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 01:14:55.472562 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457152 4427 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 01:14:55.472562 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457156 4427 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 01:14:55.472562 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457161 4427 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 01:14:55.472562 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457165 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 01:14:55.472562 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457170 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 01:14:55.472562 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457175 4427 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 01:14:55.472562 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457179 4427 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 01:14:55.472562 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457183 4427 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 01:14:55.472562 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457194 4427 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 01:14:55.472562 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457198 4427 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 01:14:55.472562 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457203 4427 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 01:14:55.472562 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457207 4427 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 01:14:55.472562 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457212 4427 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 01:14:55.473680 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457217 4427 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 01:14:55.473680 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457221 4427 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 01:14:55.473680 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457226 4427 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 01:14:55.473680 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457231 4427 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 01:14:55.473680 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457236 4427 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 01:14:55.473680 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457241 4427 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 01:14:55.473680 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457245 4427 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 01:14:55.473680 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457254 4427 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 01:14:55.473680 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457259 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 01:14:55.473680 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457263 4427 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 01:14:55.473680 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457268 4427 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 01:14:55.473680 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457272 4427 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 01:14:55.473680 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457276 4427 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 01:14:55.473680 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457281 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 01:14:55.473680 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457297 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 01:14:55.473680 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457302 4427 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 01:14:55.473680 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457306 4427 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 01:14:55.473680 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457311 4427 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 01:14:55.473680 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457315 4427 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 01:14:55.474175 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457320 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 01:14:55.474175 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457330 4427 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 01:14:55.474175 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457334 4427 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 01:14:55.474175 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457338 4427 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 01:14:55.474175 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457343 4427 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 01:14:55.474175 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457347 4427 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 01:14:55.474175 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457352 4427 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 01:14:55.474175 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457356 4427 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 01:14:55.474175 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457360 4427 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 01:14:55.474175 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457365 4427 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 01:14:55.474175 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457387 4427 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 01:14:55.474175 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457392 4427 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 01:14:55.474175 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457397 4427 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 01:14:55.474175 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457406 4427 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 01:14:55.474175 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.457467 4427 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 01:14:55.474175 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457614 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 01:14:55.474652 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457624 4427 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 01:14:55.474652 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457629 4427 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 01:14:55.474652 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457639 4427 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 01:14:55.474652 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457644 4427 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 01:14:55.474652 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457648 4427 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 01:14:55.474652 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457653 4427 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 01:14:55.474652 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457657 4427 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 01:14:55.474652 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457662 4427 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 01:14:55.474652 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457667 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 01:14:55.474652 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457671 4427 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 01:14:55.474652 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457676 4427 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 01:14:55.474652 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457681 4427 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 01:14:55.474652 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457687 4427 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 01:14:55.474652 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457692 4427 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 01:14:55.474652 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457702 4427 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 01:14:55.474652 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457707 4427 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 01:14:55.474652 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457711 4427 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 01:14:55.474652 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457716 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 01:14:55.474652 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457720 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 01:14:55.475450 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457725 4427 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 01:14:55.475450 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457729 4427 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 01:14:55.475450 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457733 4427 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 01:14:55.475450 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457738 4427 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 01:14:55.475450 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457742 4427 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 01:14:55.475450 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457746 4427 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 01:14:55.475450 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457751 4427 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 01:14:55.475450 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457755 4427 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 01:14:55.475450 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457765 4427 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 01:14:55.475450 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457770 4427 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 01:14:55.475450 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457774 4427 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 01:14:55.475450 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457779 4427 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 01:14:55.475450 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457784 4427 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 01:14:55.475450 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457789 4427 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 01:14:55.475450 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457793 4427 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 01:14:55.475450 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457801 4427 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 01:14:55.475450 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457805 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 01:14:55.475450 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457810 4427 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 01:14:55.475450 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457814 4427 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 01:14:55.475450 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457818 4427 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 01:14:55.476328 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457828 4427 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 01:14:55.476328 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457833 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 01:14:55.476328 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457838 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 01:14:55.476328 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457843 4427 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 01:14:55.476328 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457847 4427 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 01:14:55.476328 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457852 4427 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 01:14:55.476328 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457860 4427 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 01:14:55.476328 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457865 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 01:14:55.476328 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457869 4427 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 01:14:55.476328 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457873 4427 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 01:14:55.476328 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457878 4427 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 01:14:55.476328 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457882 4427 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 01:14:55.476328 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457887 4427 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 01:14:55.476328 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457896 4427 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 01:14:55.476328 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457901 4427 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 01:14:55.476328 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457905 4427 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 01:14:55.476328 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457909 4427 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 01:14:55.476328 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457913 4427 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 01:14:55.476328 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457918 4427 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 01:14:55.476993 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457925 4427 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 01:14:55.476993 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457930 4427 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 01:14:55.476993 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457935 4427 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 01:14:55.476993 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457939 4427 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 01:14:55.476993 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457944 4427 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 01:14:55.476993 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457948 4427 feature_gate.go:386] unrecognized feature gate: Example Nov 02 01:14:55.476993 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457958 4427 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 01:14:55.476993 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457962 4427 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 01:14:55.476993 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457968 4427 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 01:14:55.476993 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457973 4427 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 01:14:55.476993 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457977 4427 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 01:14:55.476993 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457981 4427 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 01:14:55.476993 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457986 4427 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 01:14:55.476993 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457990 4427 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 01:14:55.476993 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457994 4427 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 01:14:55.476993 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.457999 4427 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 01:14:55.476993 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.458003 4427 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 01:14:55.476993 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.458007 4427 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 01:14:55.476993 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.458012 4427 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 01:14:55.476993 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.458022 4427 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 01:14:55.477858 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.458027 4427 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 01:14:55.477858 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.458032 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 01:14:55.477858 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.458037 4427 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 01:14:55.477858 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.458041 4427 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 01:14:55.477858 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.458045 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 01:14:55.477858 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.458050 4427 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 01:14:55.477858 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.458054 4427 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 01:14:55.477858 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.458058 4427 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 01:14:55.477858 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.458063 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 01:14:55.477858 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.458067 4427 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 01:14:55.477858 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.458073 4427 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 01:14:55.477858 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.458085 4427 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 01:14:55.477858 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.458092 4427 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 01:14:55.477858 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.458097 4427 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 01:14:55.477858 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.458102 4427 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 01:14:55.477858 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.458107 4427 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 01:14:55.477858 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.458112 4427 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 01:14:55.477858 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.458121 4427 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 01:14:55.477858 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.458127 4427 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 01:14:55.478759 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.458132 4427 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 01:14:55.478759 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.458139 4427 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 01:14:55.478759 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.458143 4427 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 01:14:55.478759 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.458148 4427 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 01:14:55.478759 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.458157 4427 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 01:14:55.478759 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.458162 4427 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 01:14:55.478759 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.458166 4427 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 01:14:55.478759 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:14:55.458171 4427 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 01:14:55.478759 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.458537 4427 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 01:14:55.478759 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.459555 4427 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 02 01:14:55.478759 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.459613 4427 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 02 01:14:55.478759 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.459797 4427 server.go:1014] "Starting client certificate rotation" Nov 02 01:14:55.478759 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.459949 4427 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 01:14:55.478759 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.460028 4427 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 01:09:15 +0000 UTC" deadline="2028-03-29 20:11:28.927180367 +0000 UTC" Nov 02 01:14:55.478759 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.460076 4427 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12330h56m33.467107717s" Nov 02 01:14:55.478759 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.460227 4427 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 01:14:55.479573 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.461390 4427 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 01:14:55.479573 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.463162 4427 log.go:25] "Validated CRI v1 runtime API" Nov 02 01:14:55.479573 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.464657 4427 log.go:25] "Validated CRI v1 image API" Nov 02 01:14:55.479573 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.465325 4427 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 01:14:55.479573 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.466770 4427 fs.go:135] Filesystem UUIDs: map[21769c05-33cb-460b-a1d9-070bf1bcfad1:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 b22bfa15-235d-4db4-869a-1045cba4b495:/dev/nvme0n1p3] Nov 02 01:14:55.479815 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.466790 4427 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2b3edccbd7f2f3eb5e26274f63117faabff182f22a6494b0d33aa75cbb412744/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2b3edccbd7f2f3eb5e26274f63117faabff182f22a6494b0d33aa75cbb412744/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2e90faf8dd799960b8070f2406807a3587d50ac8e1a0a04ddb00976f65a9151c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2e90faf8dd799960b8070f2406807a3587d50ac8e1a0a04ddb00976f65a9151c/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3d2bf19f263abe4ecec40ed8a73450e0a6ece52303e4d7e99934da5b998f2b5b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3d2bf19f263abe4ecec40ed8a73450e0a6ece52303e4d7e99934da5b998f2b5b/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5ff16da0d1adf82e4e308b04c68653d0bb4053329ddfd88d31a058ef055e5d46/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5ff16da0d1adf82e4e308b04c68653d0bb4053329ddfd88d31a058ef055e5d46/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6bfb4f88c1ee195af16386275dfe6d1435fbf25e100db3542fc0ea3491d09763/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6bfb4f88c1ee195af16386275dfe6d1435fbf25e100db3542fc0ea3491d09763/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ae3076e51de4d109bea4f96e56ae9078cdc1e253651ff937b6412f2a75fdca73/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ae3076e51de4d109bea4f96e56ae9078cdc1e253651ff937b6412f2a75fdca73/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b57aace4925349b187d15ca591ceaa19f18b71dc50ad47e77946b14bfe8eced1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b57aace4925349b187d15ca591ceaa19f18b71dc50ad47e77946b14bfe8eced1/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b83ec72b85125f1dd4815579f8c600c42c52c752248fa07b679bdf6f9e33efcc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b83ec72b85125f1dd4815579f8c600c42c52c752248fa07b679bdf6f9e33efcc/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d434fbb46acfed32d19ce5ef28f90740f0cefe4f832dc4020078f7c956899364/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d434fbb46acfed32d19ce5ef28f90740f0cefe4f832dc4020078f7c956899364/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/edab9d76ebb11f8f2d831955285975137ed516cc47d9a4b26d8812adad4ea7e3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/edab9d76ebb11f8f2d831955285975137ed516cc47d9a4b26d8812adad4ea7e3/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f04282285db6ef418877646fb9d613c41772a44089e6d1f1c462cf9d93d4cfa5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f04282285db6ef418877646fb9d613c41772a44089e6d1f1c462cf9d93d4cfa5/userdata/shm major:0 minor:210 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f4a612f867153473c11bcddf7599b2f0e9e4df5ca9edf4801dbf9bd8e5090aa2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f4a612f867153473c11bcddf7599b2f0e9e4df5ca9edf4801dbf9bd8e5090aa2/userdata/shm major:0 minor:91 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fc108d6dd0a1a97b97982a28fea3da22b3dc06ac5e7b92c805628b335cb4b579/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fc108d6dd0a1a97b97982a28fea3da22b3dc06ac5e7b92c805628b335cb4b579/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ff3aac8d819d03f31e61d35bc7b0dbfdfa90eb9e23d42f4aeabc88adb6fa7980/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ff3aac8d819d03f31e61d35bc7b0dbfdfa90eb9e23d42f4aeabc88adb6fa7980/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/00d17dbb-cb72-4413-9559-4487c313158c/volumes/kubernetes.io~projected/kube-api-access-g9pbx:{mountpoint:/var/lib/kubelet/pods/00d17dbb-cb72-4413-9559-4487c313158c/volumes/kubernetes.io~projected/kube-api-access-g9pbx major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/16eed615-a448-4416-b06e-1c33d1fffb6d/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/16eed615-a448-4416-b06e-1c33d1fffb6d/volumes/kubernetes.io~secret/agent-certs major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/20bc48c0-b0ef-40df-a5c9-06c48c75fe20/volumes/kubernetes.io~projected/kube-api-access-9jtbq:{mountpoint:/var/lib/kubelet/pods/20bc48c0-b0ef-40df-a5c9-06c48c75fe20/volumes/kubernetes.io~projected/kube-api-access-9jtbq major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/212ace3e-7830-4810-938d-22fd7f251b25/volumes/kubernetes.io~projected/kube-api-access-7dg8x:{mountpoint:/var/lib/kubelet/pods/212ace3e-7830-4810-938d-22fd7f251b25/volumes/kubernetes.io~projected/kube-api-access-7dg8x major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/23bf37a9-9b32-47f9-a4e0-364a7273c02f/volumes/kubernetes.io~projected/kube-api-access-ntxrh:{mountpoint:/var/lib/kubelet/pods/23bf37a9-9b32-47f9-a4e0-364a7273c02f/volumes/kubernetes.io~projected/kube-api-access-ntxrh major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3917d88c-88f5-4c5c-b1f7-e0195a61f88f/volumes/kubernetes.io~projected/kube-api-access-hxrsj:{mountpoint:/var/lib/kubelet/pods/3917d88c-88f5-4c5c-b1f7-e0195a61f88f/volumes/kubernetes.io~projected/kube-api-access-hxrsj major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/41f79a83-9886-4c59-a904-328463dfe674/volumes/kubernetes.io~projected/kube-api-access-khqc7:{mountpoint:/var/lib/kubelet/pods/41f79a83-9886-4c59-a904-328463dfe674/volumes/kubernetes.io~projected/kube-api-access-khqc7 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/497c02e6-eb5a-49ed-a911-97384fb409cf/volumes/kubernetes.io~projected/kube-api-access-4p26j:{mountpoint:/var/lib/kubelet/pods/497c02e6-eb5a-49ed-a911-97384fb409cf/volumes/kubernetes.io~projected/kube-api-access-4p26j major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4a12c42f-81b2-4fde-bc01-8abe3e640957/volumes/kubernetes.io~projected/kube-api-access-bs7p2:{mountpoint:/var/lib/kubelet/pods/4a12c42f-81b2-4fde-bc01-8abe3e640957/volumes/kubernetes.io~projected/kube-api-access-bs7p2 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4c7b5fa9-d0b9-4147-bc52-bdd444a2c081/volumes/kubernetes.io~projected/kube-api-access-b98jt:{mountpoint:/var/lib/kubelet/pods/4c7b5fa9-d0b9-4147-bc52-bdd444a2c081/volumes/kubernetes.io~projected/kube-api-access-b98jt major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6a02556d-4538-478d-84cf-a4d104565c84/volumes/kubernetes.io~projected/kube-api-access-nlbbj:{mountpoint:/var/lib/kubelet/pods/6a02556d-4538-478d-84cf-a4d104565c84/volumes/kubernetes.io~projected/kube-api-access-nlbbj major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/77df90af-e163-47ea-83ad-99bc1258f3d5/volumes/kubernetes.io~projected/kube-api-access-c8tts:{mountpoint:/var/lib/kubelet/pods/77df90af-e163-47ea-83ad-99bc1258f3d5/volumes/kubernetes.io~projected/kube-api-access-c8tts major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7/volumes/kubernetes.io~projected/kube-api-access-pd76d:{mountpoint:/var/lib/kubelet/pods/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7/volumes/kubernetes.io~projected/kube-api-access-pd76d major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8cd0f6e2-e491-4b6c-943a-c1115300dcaf/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/8cd0f6e2-e491-4b6c-943a-c1115300dcaf/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8cd0f6e2-e491-4b6c-943a-c1115300dcaf/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/8cd0f6e2-e491-4b6c-943a-c1115300dcaf/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8cd0f6e2-e491-4b6c-943a-c1115300dcaf/volumes/kubernetes.io~projected/kube-api-access-s78hk:{mountpoint:/var/lib/kubelet/pods/8cd0f6e2-e491-4b6c-943a-c1115300dcaf/volumes/kubernetes.io~projected/kube-api-access-s78hk major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a75a9f45-7678-4014-92d9-93d3f5b260ba/volumes/kubernetes.io~projected/kube-api-access-987n6:{mountpoint:/var/lib/kubelet/pods/a75a9f45-7678-4014-92d9-93d3f5b260ba/volumes/kubernetes.io~projected/kube-api-access-987n6 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a75a9f45-7678-4014-92d9-93d3f5b260ba/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/a75a9f45-7678-4014-92d9-93d3f5b260ba/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/6b75dbe389ebea0c2e1f707728972753c5a1759210e3d1eb90ace80f0a896936/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/8637466f13ac2cae0ee4f775127124231f1e94c455e642201a92a495eba5d943/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/ce878694ac27668a1b994cee95dc670eab577a5013332ce6db0a1815586c5080/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/6acf5c4ff5e1093ba148d20cbc258373375033ffaeb79e5f3bac5cc86b75f95d/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/ebe0b24c1e01f117cb504eb0d3b9ac4c82a4f7cd1111b5f474da2e9669918bdb/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/3a01767ae8348201df583c175b217573f1251e54c7e6e2113fc6d392777868cd/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/a0dfa70050b40c06c8bd502309e85ad77e22f3fa3a51add9158f68a036a7e718/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/ade2db1372aa4e9280641e854c3ba905b3d1e9ff7effdda9cf4891afa29aaab1/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-131:{mountpoint:/var/lib/containers/storage/overlay/459071a8950f1ffabb541b5d365f1bc9efe5a8901f9a1f1c17c90bcf064fc721/merged major:0 minor:131 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/b12a85ab9fb80a4719684d264177c196ffad72e7d3bb94a24e9fd34171e19bc3/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/eb322179718bcb846cd025f1dcd0c59170ac97b9d8915f3ad3dee4e4437c7e89/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/270063d8f379c31adf16ecd76c1cd1f61fb3b3f3196fab946fa6bedc854172b6/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/012c2c59632dbaedd10a900b622698ed17ee8eeef0c90bca7042804c5d472536/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/b316b8133246b6ba38d8cde4631ffdce32b22991eebf1a631a87d2da7aeb40ca/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-159:{mountpoint:/var/lib/containers/storage/overlay/854bb62f8c69ff45b7e2e6d107113deb269bfad17580e1d39ac3c6c7e1ed8b74/merged major:0 minor:159 fsType:overlay blockSize:0} overlay_0-164:{mountpoint:/var/lib/containers/storage/overlay/93f0fc130fec11b8b576ec25b78cfa16a6ba67e40d6d77fccc1e875a5510cbc3/merged major:0 minor:164 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/c9b57d9d5013a41471d4152aac200ab40fe32b95d67cf87e23291732b7631bc7/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/459aedcaa7e1c5e416a6dfc1d064b136ca286d34ddd0b7bba0464ab2e7eb5b17/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/3a7e5f3b24fdf52c331add885c6ed1a82fdd396809d4057eff33f94730229028/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/0d368c18e294bd04beca2e3cfab92bd175b2b5e5b5b24e697cd9a81a2e5ca11d/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/66bf35ba738dd323a695432001c01856fe77b736de310ac746a5df7d772c7a4b/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/e8d84fafbd77c0f6a1fc8231df2436f3ba93b73b259003619a95e8d289973e9b/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-212:{mountpoint:/var/lib/containers/storage/overlay/241bf3fc4259b9bedecec1f0c9cb1398f7b5a0163275c12e982478bdbecc7caa/merged major:0 minor:212 fsType:overlay blockSize:0} overlay_0-214:{mountpoint:/var/lib/containers/storage/overlay/59d343f4340a5962f0eb94ab31b8fb58b0704bcc8a84347cb9b90d5a9de447ee/merged major:0 minor:214 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/6035d762c9fbab248749802bd42fd0bfea782d1a9ff2c046f4a0635ba95e2439/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-226:{mountpoint:/var/lib/containers/storage/overlay/18d6f3725326430b2ce4a07c299fef61e9075a6e58824fea9868a84cdb7bf1e8/merged major:0 minor:226 fsType:overlay blockSize:0} overlay_0-228:{mountpoint:/var/lib/containers/storage/overlay/04fe96db48811a118d073a150b22b15948e51e0f315c1fc6532aeeb58d08671f/merged major:0 minor:228 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/03640ff1d9fee4d844f80c6dcb6188cc5e66a9bbc237f87d0e66bc618ce76277/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/c69634301f96217b853e30171fd13fc99d64a4e04bf6e762b9fcf88e207403ed/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/b4f062a5b97636d0e4e6e8018798992406cc08acde15c6a1492272cfc5b18055/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/8e930cbb20a36738ba3d303b8e5b0598bfb4436013a46381d9e5e812b6ca112c/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/36aad4aef4f111235501c0a1f92cd05b4a56d9c542697b64c883b20f5ec25260/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/46c40c4dd51a6dd5cac3b050497705f6b0ab6a9db6d02d77ea6e959d20a95d72/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/ea1541d3e362f8fc5a43fe9d2d4f2a31ed55a8d29ffe77d6e4cb6a98b3b61ec2/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/6242a6906bca4012952a355afaf4215075bfac409c39a3b50ac02a82f865198c/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/347a659e9ec449b931548a50946c1d95082540776b775a46f2516d99f6e40413/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/9a65209ba1488fd2080f7af969eac0f960259a0643174a9ed2cf05cb47293a69/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/3680cc46088ab762dea7fabb9794a711167e105bf997cfa9625b7ad87bd96437/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/4ded6e12cd80970ea930215b1032594266b696e255decfdaf50947c8ccebdfa9/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 02 01:14:55.480035 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.472824 4427 manager.go:217] Machine: {Timestamp:2026-11-02 01:14:55.47232211 +0000 UTC m=+0.093327311 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3095973 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec270def98fc1760c6c00ee0312e9ccb SystemUUID:ec270def-98fc-1760-c6c0-0ee0312e9ccb BootID:6b66e05d-81a8-4da7-b7be-4c59ed138df8 Filesystems:[{Device:/var/lib/kubelet/pods/212ace3e-7830-4810-938d-22fd7f251b25/volumes/kubernetes.io~projected/kube-api-access-7dg8x DeviceMajor:0 DeviceMinor:223 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7/volumes/kubernetes.io~projected/kube-api-access-pd76d DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8cd0f6e2-e491-4b6c-943a-c1115300dcaf/volumes/kubernetes.io~projected/kube-api-access-s78hk DeviceMajor:0 DeviceMinor:62 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-164 DeviceMajor:0 DeviceMinor:164 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a75a9f45-7678-4014-92d9-93d3f5b260ba/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fc108d6dd0a1a97b97982a28fea3da22b3dc06ac5e7b92c805628b335cb4b579/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-226 DeviceMajor:0 DeviceMinor:226 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-214 DeviceMajor:0 DeviceMinor:214 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-228 DeviceMajor:0 DeviceMinor:228 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2b3edccbd7f2f3eb5e26274f63117faabff182f22a6494b0d33aa75cbb412744/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3d2bf19f263abe4ecec40ed8a73450e0a6ece52303e4d7e99934da5b998f2b5b/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6a02556d-4538-478d-84cf-a4d104565c84/volumes/kubernetes.io~projected/kube-api-access-nlbbj DeviceMajor:0 DeviceMinor:65 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5ff16da0d1adf82e4e308b04c68653d0bb4053329ddfd88d31a058ef055e5d46/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d434fbb46acfed32d19ce5ef28f90740f0cefe4f832dc4020078f7c956899364/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/16eed615-a448-4416-b06e-1c33d1fffb6d/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:64 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b83ec72b85125f1dd4815579f8c600c42c52c752248fa07b679bdf6f9e33efcc/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/20bc48c0-b0ef-40df-a5c9-06c48c75fe20/volumes/kubernetes.io~projected/kube-api-access-9jtbq DeviceMajor:0 DeviceMinor:61 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/77df90af-e163-47ea-83ad-99bc1258f3d5/volumes/kubernetes.io~projected/kube-api-access-c8tts DeviceMajor:0 DeviceMinor:63 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/41f79a83-9886-4c59-a904-328463dfe674/volumes/kubernetes.io~projected/kube-api-access-khqc7 DeviceMajor:0 DeviceMinor:67 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-212 DeviceMajor:0 DeviceMinor:212 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ae3076e51de4d109bea4f96e56ae9078cdc1e253651ff937b6412f2a75fdca73/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f4a612f867153473c11bcddf7599b2f0e9e4df5ca9edf4801dbf9bd8e5090aa2/userdata/shm DeviceMajor:0 DeviceMinor:91 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-131 DeviceMajor:0 DeviceMinor:131 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:206 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b57aace4925349b187d15ca591ceaa19f18b71dc50ad47e77946b14bfe8eced1/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f04282285db6ef418877646fb9d613c41772a44089e6d1f1c462cf9d93d4cfa5/userdata/shm DeviceMajor:0 DeviceMinor:210 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/8cd0f6e2-e491-4b6c-943a-c1115300dcaf/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/edab9d76ebb11f8f2d831955285975137ed516cc47d9a4b26d8812adad4ea7e3/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/00d17dbb-cb72-4413-9559-4487c313158c/volumes/kubernetes.io~projected/kube-api-access-g9pbx DeviceMajor:0 DeviceMinor:197 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-159 DeviceMajor:0 DeviceMinor:159 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ff3aac8d819d03f31e61d35bc7b0dbfdfa90eb9e23d42f4aeabc88adb6fa7980/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/23bf37a9-9b32-47f9-a4e0-364a7273c02f/volumes/kubernetes.io~projected/kube-api-access-ntxrh DeviceMajor:0 DeviceMinor:218 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2e90faf8dd799960b8070f2406807a3587d50ac8e1a0a04ddb00976f65a9151c/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4c7b5fa9-d0b9-4147-bc52-bdd444a2c081/volumes/kubernetes.io~projected/kube-api-access-b98jt DeviceMajor:0 DeviceMinor:66 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3917d88c-88f5-4c5c-b1f7-e0195a61f88f/volumes/kubernetes.io~projected/kube-api-access-hxrsj DeviceMajor:0 DeviceMinor:207 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/497c02e6-eb5a-49ed-a911-97384fb409cf/volumes/kubernetes.io~projected/kube-api-access-4p26j DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4a12c42f-81b2-4fde-bc01-8abe3e640957/volumes/kubernetes.io~projected/kube-api-access-bs7p2 DeviceMajor:0 DeviceMinor:60 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6bfb4f88c1ee195af16386275dfe6d1435fbf25e100db3542fc0ea3491d09763/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a75a9f45-7678-4014-92d9-93d3f5b260ba/volumes/kubernetes.io~projected/kube-api-access-987n6 DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8cd0f6e2-e491-4b6c-943a-c1115300dcaf/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e6:7a:af:b1 Speed:0 Mtu:9001} {Name:br-int MacAddress:16:cd:99:96:88:6b Speed:0 Mtu:8901} {Name:d434fbb46acfed3 MacAddress:ae:a8:7d:2a:f1:55 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:e6:7a:af:b1 Speed:0 Mtu:9001} {Name:f04282285db6ef4 MacAddress:a2:bb:05:84:e5:2f Speed:10000 Mtu:8901} {Name:fc108d6dd0a1a97 MacAddress:02:65:8e:f3:8c:cd Speed:10000 Mtu:8901} {Name:ff3aac8d819d03f MacAddress:d6:1a:7a:a2:b1:af Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:fe:59:24:f9:99:f5 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:46:79:53:ff:38:00 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 01:14:55.480035 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.473061 4427 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 01:14:55.480035 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.473104 4427 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 01:14:55.480035 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.473417 4427 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 01:14:55.480035 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.473440 4427 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-130.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 01:14:55.480035 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.473565 4427 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 01:14:55.480035 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.473572 4427 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 01:14:55.480035 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.473583 4427 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 01:14:55.480035 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.473602 4427 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 01:14:55.480035 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.473705 4427 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 01:14:55.480035 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.473857 4427 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 01:14:55.480035 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.473967 4427 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 01:14:55.480035 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.474002 4427 kubelet.go:493] "Attempting to sync node with API server" Nov 02 01:14:55.480035 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.474017 4427 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 01:14:55.480035 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.474028 4427 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 01:14:55.480035 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.474038 4427 kubelet.go:405] "Adding apiserver pod source" Nov 02 01:14:55.480035 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.474046 4427 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 01:14:55.480035 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.475561 4427 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 01:14:55.480035 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.475583 4427 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 01:14:55.480035 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.476561 4427 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 01:14:55.480035 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.479888 4427 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 01:14:55.480035 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.479903 4427 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 01:14:55.480035 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.480026 4427 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 02 01:14:55.481297 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.480294 4427 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 01:14:55.481297 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.480336 4427 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 01:14:55.481297 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.480474 4427 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 01:14:55.481297 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.480502 4427 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 01:14:55.481297 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.480512 4427 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 01:14:55.481297 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.480522 4427 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 01:14:55.481297 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.480532 4427 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 01:14:55.481297 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.480541 4427 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 01:14:55.481297 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.480550 4427 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 01:14:55.481297 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.480561 4427 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 01:14:55.481297 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.480598 4427 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 01:14:55.481297 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.480609 4427 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 01:14:55.481297 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.480624 4427 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 01:14:55.481297 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.480638 4427 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 01:14:55.481297 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.480673 4427 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 01:14:55.481297 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.480683 4427 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 01:14:55.484525 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.484508 4427 server.go:1263] "Started kubelet" Nov 02 01:14:55.484605 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.484564 4427 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 01:14:55.485011 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.484659 4427 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 01:14:55.485011 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.484750 4427 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 01:14:55.485149 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.485071 4427 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 01:14:55.485156 ip-10-0-136-130 systemd[1]: Started Kubernetes Kubelet. Nov 02 01:14:55.486332 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.486314 4427 server.go:317] "Adding debug handlers to kubelet server" Nov 02 01:14:55.488886 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.487623 4427 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 01:14:55.488886 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.487787 4427 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 01:14:55.488886 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.487964 4427 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 01:09:17 +0000 UTC" deadline="2028-08-04 21:48:32.777548764 +0000 UTC" Nov 02 01:14:55.488886 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.488002 4427 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15404h33m37.289549938s" Nov 02 01:14:55.488886 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:14:55.488331 4427 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-130.us-east-2.compute.internal\" not found" Nov 02 01:14:55.488886 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.488538 4427 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 01:14:55.488886 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.488596 4427 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 01:14:55.488886 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.488789 4427 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 01:14:55.492401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.490054 4427 factory.go:153] Registering CRI-O factory Nov 02 01:14:55.492401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.490074 4427 factory.go:223] Registration of the crio container factory successfully Nov 02 01:14:55.492401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.490137 4427 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 01:14:55.492401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.490146 4427 factory.go:55] Registering systemd factory Nov 02 01:14:55.492401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.490153 4427 factory.go:223] Registration of the systemd container factory successfully Nov 02 01:14:55.492401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.490196 4427 factory.go:103] Registering Raw factory Nov 02 01:14:55.492401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.490230 4427 manager.go:1199] Started watching for new ooms in manager Nov 02 01:14:55.492401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.490661 4427 manager.go:319] Starting recovery of all containers Nov 02 01:14:55.492401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.491669 4427 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="212ace3e-7830-4810-938d-22fd7f251b25" volumeName="kubernetes.io/projected/212ace3e-7830-4810-938d-22fd7f251b25-kube-api-access-7dg8x" seLinuxMountContext="" Nov 02 01:14:55.492401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.491720 4427 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00d17dbb-cb72-4413-9559-4487c313158c" volumeName="kubernetes.io/projected/00d17dbb-cb72-4413-9559-4487c313158c-kube-api-access-g9pbx" seLinuxMountContext="" Nov 02 01:14:55.492401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.491754 4427 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cd0f6e2-e491-4b6c-943a-c1115300dcaf" volumeName="kubernetes.io/empty-dir/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-tmp" seLinuxMountContext="" Nov 02 01:14:55.492401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.491781 4427 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a75a9f45-7678-4014-92d9-93d3f5b260ba" volumeName="kubernetes.io/projected/a75a9f45-7678-4014-92d9-93d3f5b260ba-kube-api-access-987n6" seLinuxMountContext="" Nov 02 01:14:55.492401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.491808 4427 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00d17dbb-cb72-4413-9559-4487c313158c" volumeName="kubernetes.io/configmap/00d17dbb-cb72-4413-9559-4487c313158c-config-volume" seLinuxMountContext="" Nov 02 01:14:55.492401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.491824 4427 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="23bf37a9-9b32-47f9-a4e0-364a7273c02f" volumeName="kubernetes.io/projected/23bf37a9-9b32-47f9-a4e0-364a7273c02f-kube-api-access-ntxrh" seLinuxMountContext="" Nov 02 01:14:55.492401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.491846 4427 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7" volumeName="kubernetes.io/secret/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-ovn-node-metrics-cert" seLinuxMountContext="" Nov 02 01:14:55.492401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.491865 4427 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a75a9f45-7678-4014-92d9-93d3f5b260ba" volumeName="kubernetes.io/secret/a75a9f45-7678-4014-92d9-93d3f5b260ba-cert" seLinuxMountContext="" Nov 02 01:14:55.492401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.491883 4427 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="77df90af-e163-47ea-83ad-99bc1258f3d5" volumeName="kubernetes.io/projected/77df90af-e163-47ea-83ad-99bc1258f3d5-kube-api-access-c8tts" seLinuxMountContext="" Nov 02 01:14:55.492401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.491906 4427 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16eed615-a448-4416-b06e-1c33d1fffb6d" volumeName="kubernetes.io/configmap/16eed615-a448-4416-b06e-1c33d1fffb6d-konnectivity-ca" seLinuxMountContext="" Nov 02 01:14:55.492401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.491918 4427 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3917d88c-88f5-4c5c-b1f7-e0195a61f88f" volumeName="kubernetes.io/projected/3917d88c-88f5-4c5c-b1f7-e0195a61f88f-kube-api-access-hxrsj" seLinuxMountContext="" Nov 02 01:14:55.492401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.491931 4427 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41f79a83-9886-4c59-a904-328463dfe674" volumeName="kubernetes.io/configmap/41f79a83-9886-4c59-a904-328463dfe674-serviceca" seLinuxMountContext="" Nov 02 01:14:55.492401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.491944 4427 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4a12c42f-81b2-4fde-bc01-8abe3e640957" volumeName="kubernetes.io/configmap/4a12c42f-81b2-4fde-bc01-8abe3e640957-multus-daemon-config" seLinuxMountContext="" Nov 02 01:14:55.492401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.491957 4427 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a02556d-4538-478d-84cf-a4d104565c84" volumeName="kubernetes.io/configmap/6a02556d-4538-478d-84cf-a4d104565c84-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 02 01:14:55.492401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.491974 4427 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00d17dbb-cb72-4413-9559-4487c313158c" volumeName="kubernetes.io/empty-dir/00d17dbb-cb72-4413-9559-4487c313158c-tmp-dir" seLinuxMountContext="" Nov 02 01:14:55.492401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.491986 4427 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="23bf37a9-9b32-47f9-a4e0-364a7273c02f" volumeName="kubernetes.io/empty-dir/23bf37a9-9b32-47f9-a4e0-364a7273c02f-data-volume" seLinuxMountContext="" Nov 02 01:14:55.492401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.491999 4427 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4a12c42f-81b2-4fde-bc01-8abe3e640957" volumeName="kubernetes.io/projected/4a12c42f-81b2-4fde-bc01-8abe3e640957-kube-api-access-bs7p2" seLinuxMountContext="" Nov 02 01:14:55.492401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.492011 4427 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cd0f6e2-e491-4b6c-943a-c1115300dcaf" volumeName="kubernetes.io/projected/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-kube-api-access-s78hk" seLinuxMountContext="" Nov 02 01:14:55.492401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.492034 4427 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="497c02e6-eb5a-49ed-a911-97384fb409cf" volumeName="kubernetes.io/projected/497c02e6-eb5a-49ed-a911-97384fb409cf-kube-api-access-4p26j" seLinuxMountContext="" Nov 02 01:14:55.492401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.492047 4427 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="497c02e6-eb5a-49ed-a911-97384fb409cf" volumeName="kubernetes.io/configmap/497c02e6-eb5a-49ed-a911-97384fb409cf-iptables-alerter-script" seLinuxMountContext="" Nov 02 01:14:55.492401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.492058 4427 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a02556d-4538-478d-84cf-a4d104565c84" volumeName="kubernetes.io/configmap/6a02556d-4538-478d-84cf-a4d104565c84-cni-sysctl-allowlist" seLinuxMountContext="" Nov 02 01:14:55.492401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.492070 4427 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a02556d-4538-478d-84cf-a4d104565c84" volumeName="kubernetes.io/projected/6a02556d-4538-478d-84cf-a4d104565c84-kube-api-access-nlbbj" seLinuxMountContext="" Nov 02 01:14:55.492401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.492086 4427 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7" volumeName="kubernetes.io/configmap/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-ovnkube-config" seLinuxMountContext="" Nov 02 01:14:55.492401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.492102 4427 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7" volumeName="kubernetes.io/configmap/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-ovnkube-script-lib" seLinuxMountContext="" Nov 02 01:14:55.492401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.492118 4427 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cd0f6e2-e491-4b6c-943a-c1115300dcaf" volumeName="kubernetes.io/empty-dir/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-etc-tuned" seLinuxMountContext="" Nov 02 01:14:55.492401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.492130 4427 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4a12c42f-81b2-4fde-bc01-8abe3e640957" volumeName="kubernetes.io/configmap/4a12c42f-81b2-4fde-bc01-8abe3e640957-cni-binary-copy" seLinuxMountContext="" Nov 02 01:14:55.492401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.492142 4427 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c7b5fa9-d0b9-4147-bc52-bdd444a2c081" volumeName="kubernetes.io/projected/4c7b5fa9-d0b9-4147-bc52-bdd444a2c081-kube-api-access-b98jt" seLinuxMountContext="" Nov 02 01:14:55.492401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.492170 4427 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="77df90af-e163-47ea-83ad-99bc1258f3d5" volumeName="kubernetes.io/empty-dir/77df90af-e163-47ea-83ad-99bc1258f3d5-tmp-dir" seLinuxMountContext="" Nov 02 01:14:55.492401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.492204 4427 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7" volumeName="kubernetes.io/configmap/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-env-overrides" seLinuxMountContext="" Nov 02 01:14:55.492401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.492220 4427 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074" volumeName="kubernetes.io/secret/9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074-original-pull-secret" seLinuxMountContext="" Nov 02 01:14:55.492401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.492232 4427 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a02556d-4538-478d-84cf-a4d104565c84" volumeName="kubernetes.io/configmap/6a02556d-4538-478d-84cf-a4d104565c84-cni-binary-copy" seLinuxMountContext="" Nov 02 01:14:55.492401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.492244 4427 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7" volumeName="kubernetes.io/projected/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-kube-api-access-pd76d" seLinuxMountContext="" Nov 02 01:14:55.492401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.492256 4427 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16eed615-a448-4416-b06e-1c33d1fffb6d" volumeName="kubernetes.io/secret/16eed615-a448-4416-b06e-1c33d1fffb6d-agent-certs" seLinuxMountContext="" Nov 02 01:14:55.492401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.492269 4427 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20bc48c0-b0ef-40df-a5c9-06c48c75fe20" volumeName="kubernetes.io/projected/20bc48c0-b0ef-40df-a5c9-06c48c75fe20-kube-api-access-9jtbq" seLinuxMountContext="" Nov 02 01:14:55.492401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.492297 4427 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41f79a83-9886-4c59-a904-328463dfe674" volumeName="kubernetes.io/projected/41f79a83-9886-4c59-a904-328463dfe674-kube-api-access-khqc7" seLinuxMountContext="" Nov 02 01:14:55.492401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.492310 4427 reconstruct.go:97] "Volume reconstruction finished" Nov 02 01:14:55.492401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.492317 4427 reconciler.go:29] "Reconciler: start to sync state" Nov 02 01:14:55.495852 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.495831 4427 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 01:14:55.496109 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.496094 4427 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 01:14:55.497684 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.497664 4427 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 01:14:55.502075 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.502055 4427 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 02 01:14:55.524037 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.523986 4427 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 01:14:55.525673 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.525651 4427 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 01:14:55.525673 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.525677 4427 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 01:14:55.525852 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.525698 4427 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 01:14:55.525852 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:14:55.525730 4427 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 02 01:14:55.535091 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.535064 4427 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 01:14:55.537826 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.537781 4427 generic.go:356] "Generic (PLEG): container finished" podID="b96c82052e4f897819ee4482b034eaf4" containerID="129512017e109b75bcb5812581d26004e489e69f2b4523cf7117fd9021625a78" exitCode=0 Nov 02 01:14:55.551812 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.551785 4427 generic.go:356] "Generic (PLEG): container finished" podID="6a02556d-4538-478d-84cf-a4d104565c84" containerID="1c556fc4305e0e0a9fa951050f19cb59acd55fc59ff711195dd43234dce73b0d" exitCode=0 Nov 02 01:14:55.551812 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.551809 4427 generic.go:356] "Generic (PLEG): container finished" podID="6a02556d-4538-478d-84cf-a4d104565c84" containerID="2d41b2e6fc23f326705993f0fe16ed0a0bca86edd1030b74691bf2e5f3968f81" exitCode=0 Nov 02 01:14:55.551812 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.551817 4427 generic.go:356] "Generic (PLEG): container finished" podID="6a02556d-4538-478d-84cf-a4d104565c84" containerID="6807bcba93370507b7b9c14d68ae9269196d027dc71ac2b2d16b2bc93aa10005" exitCode=0 Nov 02 01:14:55.552052 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.551828 4427 generic.go:356] "Generic (PLEG): container finished" podID="6a02556d-4538-478d-84cf-a4d104565c84" containerID="bfa0e14112d985d99c151707104fd44b4f23b2480698fadfd4d23375f951165e" exitCode=0 Nov 02 01:14:55.552052 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.551836 4427 generic.go:356] "Generic (PLEG): container finished" podID="6a02556d-4538-478d-84cf-a4d104565c84" containerID="89bdd479f0394295448197347410602769e38605c6035ab1a86bcdcb2c5ffcd8" exitCode=0 Nov 02 01:14:55.552052 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.551843 4427 generic.go:356] "Generic (PLEG): container finished" podID="6a02556d-4538-478d-84cf-a4d104565c84" containerID="d256879fb5ca93312bd104d0a61b5c405f0fe8bed03c321c0251d1626f524d66" exitCode=0 Nov 02 01:14:55.557417 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.557396 4427 manager.go:324] Recovery completed Nov 02 01:14:55.571187 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.571126 4427 cpu_manager.go:226] "Starting" policy="none" Nov 02 01:14:55.571187 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.571149 4427 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 01:14:55.571187 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.571170 4427 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 01:14:55.571368 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.571355 4427 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 02 01:14:55.571441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.571389 4427 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 02 01:14:55.571441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.571415 4427 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 02 01:14:55.571441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.571424 4427 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 02 01:14:55.571441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.571431 4427 policy_none.go:50] "Start" Nov 02 01:14:55.571599 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.571451 4427 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 01:14:55.571599 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.571464 4427 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 01:14:55.571702 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.571686 4427 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 02 01:14:55.571751 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.571705 4427 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 02 01:14:55.571751 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.571713 4427 policy_none.go:44] "Start" Nov 02 01:14:55.576412 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.576364 4427 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 01:14:55.579909 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.579895 4427 manager.go:341] "Starting Device Plugin manager" Nov 02 01:14:55.580483 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:14:55.579920 4427 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 01:14:55.580530 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.580486 4427 server.go:87] "Starting device plugin registration server" Nov 02 01:14:55.580738 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.580726 4427 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 01:14:55.580768 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.580741 4427 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 01:14:55.580870 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.580853 4427 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 01:14:55.581022 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.580959 4427 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 01:14:55.581022 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.580999 4427 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 01:14:55.581022 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.581005 4427 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 01:14:55.581162 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.581057 4427 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T01:14:55.580988321Z","UUID":"b84254a9-e54d-4908-adcc-89aea3231c89","Handler":null,"Name":"","Endpoint":""} Nov 02 01:14:55.582065 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:14:55.582047 4427 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 01:14:55.582396 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.582365 4427 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 01:14:55.582484 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.582409 4427 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 01:14:55.626045 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.625995 4427 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-130.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-130.us-east-2.compute.internal"] Nov 02 01:14:55.626652 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.626602 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-130.us-east-2.compute.internal" event={"ID":"b96c82052e4f897819ee4482b034eaf4","Type":"ContainerStarted","Data":"1d12683f13340c1587b03fe580d5dd0be379c9b659bde5e6b9fbf1e4122d14ee"} Nov 02 01:14:55.626652 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.626655 4427 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-130.us-east-2.compute.internal" Nov 02 01:14:55.626874 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.626659 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-130.us-east-2.compute.internal" event={"ID":"b96c82052e4f897819ee4482b034eaf4","Type":"ContainerDied","Data":"129512017e109b75bcb5812581d26004e489e69f2b4523cf7117fd9021625a78"} Nov 02 01:14:55.626874 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.626683 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-130.us-east-2.compute.internal" event={"ID":"b96c82052e4f897819ee4482b034eaf4","Type":"ContainerStarted","Data":"3d2bf19f263abe4ecec40ed8a73450e0a6ece52303e4d7e99934da5b998f2b5b"} Nov 02 01:14:55.626874 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.626740 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-130.us-east-2.compute.internal" event={"ID":"dabfce80149e41ad659d129c965945bf","Type":"ContainerStarted","Data":"4bdf24dc6473df78ee8e7e673408f85f99f49c91250776dd1af836e084343c4f"} Nov 02 01:14:55.626874 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.626759 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-130.us-east-2.compute.internal" event={"ID":"dabfce80149e41ad659d129c965945bf","Type":"ContainerStarted","Data":"2e90faf8dd799960b8070f2406807a3587d50ac8e1a0a04ddb00976f65a9151c"} Nov 02 01:14:55.627079 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.626893 4427 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-130.us-east-2.compute.internal" Nov 02 01:14:55.638877 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.638857 4427 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 01:14:55.639000 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:14:55.638916 4427 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-130.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-130.us-east-2.compute.internal" Nov 02 01:14:55.639606 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.639587 4427 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 01:14:55.639648 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:14:55.639631 4427 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-130.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-130.us-east-2.compute.internal" Nov 02 01:14:55.681741 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.681714 4427 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 01:14:55.683460 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.683443 4427 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-130.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 01:14:55.683535 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.683474 4427 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-130.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 01:14:55.683535 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.683489 4427 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-130.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 02 01:14:55.683535 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.683529 4427 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-130.us-east-2.compute.internal" Nov 02 01:14:55.695789 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.695771 4427 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-136-130.us-east-2.compute.internal" Nov 02 01:14:55.695835 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.695828 4427 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-130.us-east-2.compute.internal" Nov 02 01:14:55.703072 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.703050 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b96c82052e4f897819ee4482b034eaf4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-130.us-east-2.compute.internal\" (UID: \"b96c82052e4f897819ee4482b034eaf4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-130.us-east-2.compute.internal" Nov 02 01:14:55.703189 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.703077 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b96c82052e4f897819ee4482b034eaf4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-130.us-east-2.compute.internal\" (UID: \"b96c82052e4f897819ee4482b034eaf4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-130.us-east-2.compute.internal" Nov 02 01:14:55.703189 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.703094 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dabfce80149e41ad659d129c965945bf-config\") pod \"kube-apiserver-proxy-ip-10-0-136-130.us-east-2.compute.internal\" (UID: \"dabfce80149e41ad659d129c965945bf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-130.us-east-2.compute.internal" Nov 02 01:14:55.803870 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.803829 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b96c82052e4f897819ee4482b034eaf4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-130.us-east-2.compute.internal\" (UID: \"b96c82052e4f897819ee4482b034eaf4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-130.us-east-2.compute.internal" Nov 02 01:14:55.803870 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.803861 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b96c82052e4f897819ee4482b034eaf4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-130.us-east-2.compute.internal\" (UID: \"b96c82052e4f897819ee4482b034eaf4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-130.us-east-2.compute.internal" Nov 02 01:14:55.804080 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.803915 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b96c82052e4f897819ee4482b034eaf4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-130.us-east-2.compute.internal\" (UID: \"b96c82052e4f897819ee4482b034eaf4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-130.us-east-2.compute.internal" Nov 02 01:14:55.804080 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.803940 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dabfce80149e41ad659d129c965945bf-config\") pod \"kube-apiserver-proxy-ip-10-0-136-130.us-east-2.compute.internal\" (UID: \"dabfce80149e41ad659d129c965945bf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-130.us-east-2.compute.internal" Nov 02 01:14:55.804080 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.803977 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dabfce80149e41ad659d129c965945bf-config\") pod \"kube-apiserver-proxy-ip-10-0-136-130.us-east-2.compute.internal\" (UID: \"dabfce80149e41ad659d129c965945bf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-130.us-east-2.compute.internal" Nov 02 01:14:55.804080 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:55.804014 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b96c82052e4f897819ee4482b034eaf4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-130.us-east-2.compute.internal\" (UID: \"b96c82052e4f897819ee4482b034eaf4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-130.us-east-2.compute.internal" Nov 02 01:14:56.474706 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.474667 4427 apiserver.go:52] "Watching apiserver" Nov 02 01:14:56.484234 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.484211 4427 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 01:14:56.484623 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.484606 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-qwzn4","openshift-kube-storage-version-migrator/migrator-66df5d496b-ctstx","kube-system/konnectivity-agent-vcsls","openshift-network-diagnostics/network-check-target-fdptg","kube-system/global-pull-secret-syncer-kl8v7","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj5fn","openshift-multus/multus-27jtx","openshift-image-registry/node-ca-bgczm","openshift-ingress-canary/ingress-canary-pn4kl","openshift-multus/network-metrics-daemon-nkqjm","kube-system/kube-apiserver-proxy-ip-10-0-136-130.us-east-2.compute.internal","openshift-dns/dns-default-xcm6b","openshift-cluster-node-tuning-operator/tuned-pqmz2","openshift-insights/insights-runtime-extractor-gs62j","openshift-network-operator/iptables-alerter-phtn9","openshift-ovn-kubernetes/ovnkube-node-z8bzg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-130.us-east-2.compute.internal","openshift-multus/multus-additional-cni-plugins-snk78"] Nov 02 01:14:56.487921 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.487900 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkqjm" Nov 02 01:14:56.488712 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.488697 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xcm6b" Nov 02 01:14:56.489440 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.489369 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gs62j" Nov 02 01:14:56.489729 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.489708 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 01:14:56.489834 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.489798 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 01:14:56.489834 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.489802 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 01:14:56.489952 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.489849 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 01:14:56.489952 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.489903 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 01:14:56.489952 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.489902 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 01:14:56.489952 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.489923 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 01:14:56.490129 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.489999 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 01:14:56.490289 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.490269 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 01:14:56.490340 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.490274 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 01:14:56.490681 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.490662 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 01:14:56.491334 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.491300 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 01:14:56.491334 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.491326 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vlhpz\"" Nov 02 01:14:56.491997 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.491977 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-85fmt\"" Nov 02 01:14:56.492121 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.492069 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-75lmf\"" Nov 02 01:14:56.492185 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.492148 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 01:14:56.492362 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.492336 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-tmdq2\"" Nov 02 01:14:56.492480 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.492418 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 01:14:56.492650 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.492633 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-g7hgp\"" Nov 02 01:14:56.492787 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.492771 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5kn9m\"" Nov 02 01:14:56.492851 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.492835 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8rk5x\"" Nov 02 01:14:56.492923 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.492907 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7pl9l\"" Nov 02 01:14:56.492968 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.492951 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-67z2p\"" Nov 02 01:14:56.493237 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.493221 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wfrhp\"" Nov 02 01:14:56.493281 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.493251 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-87cgr\"" Nov 02 01:14:56.493337 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.493299 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 01:14:56.493337 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.493309 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 01:14:56.493450 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.493341 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-7fmpb\"" Nov 02 01:14:56.493691 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.493675 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 01:14:56.493739 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.493697 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 01:14:56.493781 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.493739 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hdjbk\"" Nov 02 01:14:56.493968 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.493953 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 01:14:56.494011 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.493967 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-rcvqj\"" Nov 02 01:14:56.494062 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.494036 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 01:14:56.494348 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.494331 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-g6rtc\"" Nov 02 01:14:56.494424 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.494398 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 01:14:56.494605 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.494587 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 01:14:56.494704 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.494687 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 01:14:56.494749 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.494717 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 01:14:56.494749 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.494744 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 01:14:56.494853 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.494839 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 01:14:56.495035 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.495019 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 01:14:56.495081 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.495061 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 01:14:56.495138 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.495123 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 01:14:56.495182 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.495167 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 01:14:56.495417 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.495400 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 02 01:14:56.495479 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.495459 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 01:14:56.495522 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.495495 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 01:14:56.495737 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.495715 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 01:14:56.495863 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.495834 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 01:14:56.495983 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.495966 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-ck74h\"" Nov 02 01:14:56.496302 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.496285 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 01:14:56.496413 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.496286 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 01:14:56.496413 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.496300 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 02 01:14:56.496597 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.496323 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 01:14:56.496597 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.496324 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 01:14:56.496597 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.496324 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 01:14:56.496597 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.496493 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 01:14:56.508085 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.508059 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-lib-modules\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:56.508183 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.508087 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074-dbus\") pod \"global-pull-secret-syncer-kl8v7\" (UID: \"9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074\") " pod="kube-system/global-pull-secret-syncer-kl8v7" Nov 02 01:14:56.508183 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.508104 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6a02556d-4538-478d-84cf-a4d104565c84-cnibin\") pod \"multus-additional-cni-plugins-snk78\" (UID: \"6a02556d-4538-478d-84cf-a4d104565c84\") " pod="openshift-multus/multus-additional-cni-plugins-snk78" Nov 02 01:14:56.508183 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.508124 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-systemd-units\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:56.508183 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.508167 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-run\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:56.508342 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.508205 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/00d17dbb-cb72-4413-9559-4487c313158c-config-volume\") pod \"dns-default-xcm6b\" (UID: \"00d17dbb-cb72-4413-9559-4487c313158c\") " pod="openshift-dns/dns-default-xcm6b" Nov 02 01:14:56.508342 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.508251 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/20bc48c0-b0ef-40df-a5c9-06c48c75fe20-sys-fs\") pod \"aws-ebs-csi-driver-node-hj5fn\" (UID: \"20bc48c0-b0ef-40df-a5c9-06c48c75fe20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj5fn" Nov 02 01:14:56.508342 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.508313 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4a12c42f-81b2-4fde-bc01-8abe3e640957-multus-daemon-config\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:56.508473 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.508363 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-host-run-multus-certs\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:56.508559 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.508487 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/00d17dbb-cb72-4413-9559-4487c313158c-config-volume\") pod \"dns-default-xcm6b\" (UID: \"00d17dbb-cb72-4413-9559-4487c313158c\") " pod="openshift-dns/dns-default-xcm6b" Nov 02 01:14:56.508559 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.508506 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a02556d-4538-478d-84cf-a4d104565c84-system-cni-dir\") pod \"multus-additional-cni-plugins-snk78\" (UID: \"6a02556d-4538-478d-84cf-a4d104565c84\") " pod="openshift-multus/multus-additional-cni-plugins-snk78" Nov 02 01:14:56.508642 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.508588 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6a02556d-4538-478d-84cf-a4d104565c84-tuning-conf-dir\") pod \"multus-additional-cni-plugins-snk78\" (UID: \"6a02556d-4538-478d-84cf-a4d104565c84\") " pod="openshift-multus/multus-additional-cni-plugins-snk78" Nov 02 01:14:56.508642 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.508632 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-host-run-netns\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:56.508708 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.508670 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-host-var-lib-kubelet\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:56.508708 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.508691 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4a12c42f-81b2-4fde-bc01-8abe3e640957-multus-daemon-config\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:56.508708 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.508698 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-multus-conf-dir\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:56.508849 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.508737 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs7p2\" (UniqueName: \"kubernetes.io/projected/4a12c42f-81b2-4fde-bc01-8abe3e640957-kube-api-access-bs7p2\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:56.508849 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.508770 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/23bf37a9-9b32-47f9-a4e0-364a7273c02f-crio-socket\") pod \"insights-runtime-extractor-gs62j\" (UID: \"23bf37a9-9b32-47f9-a4e0-364a7273c02f\") " pod="openshift-insights/insights-runtime-extractor-gs62j" Nov 02 01:14:56.508849 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.508796 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-ovnkube-config\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:56.508849 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.508819 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-env-overrides\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:56.509023 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.508848 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd76d\" (UniqueName: \"kubernetes.io/projected/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-kube-api-access-pd76d\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:56.509023 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.508873 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-multus-socket-dir-parent\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:56.509023 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.508899 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6a02556d-4538-478d-84cf-a4d104565c84-cni-binary-copy\") pod \"multus-additional-cni-plugins-snk78\" (UID: \"6a02556d-4538-478d-84cf-a4d104565c84\") " pod="openshift-multus/multus-additional-cni-plugins-snk78" Nov 02 01:14:56.509023 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.508926 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6a02556d-4538-478d-84cf-a4d104565c84-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-snk78\" (UID: \"6a02556d-4538-478d-84cf-a4d104565c84\") " pod="openshift-multus/multus-additional-cni-plugins-snk78" Nov 02 01:14:56.509023 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.508952 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlbbj\" (UniqueName: \"kubernetes.io/projected/6a02556d-4538-478d-84cf-a4d104565c84-kube-api-access-nlbbj\") pod \"multus-additional-cni-plugins-snk78\" (UID: \"6a02556d-4538-478d-84cf-a4d104565c84\") " pod="openshift-multus/multus-additional-cni-plugins-snk78" Nov 02 01:14:56.509023 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.508974 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-env-overrides\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:56.509023 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.508995 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-ovnkube-config\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:56.509023 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.508998 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8tts\" (UniqueName: \"kubernetes.io/projected/77df90af-e163-47ea-83ad-99bc1258f3d5-kube-api-access-c8tts\") pod \"node-resolver-qwzn4\" (UID: \"77df90af-e163-47ea-83ad-99bc1258f3d5\") " pod="openshift-dns/node-resolver-qwzn4" Nov 02 01:14:56.509346 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.509071 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-run-systemd\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:56.509346 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.509135 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/497c02e6-eb5a-49ed-a911-97384fb409cf-host-slash\") pod \"iptables-alerter-phtn9\" (UID: \"497c02e6-eb5a-49ed-a911-97384fb409cf\") " pod="openshift-network-operator/iptables-alerter-phtn9" Nov 02 01:14:56.509346 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.509162 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-etc-sysctl-conf\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:56.509346 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.509195 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-multus-cni-dir\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:56.509346 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.509233 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/16eed615-a448-4416-b06e-1c33d1fffb6d-agent-certs\") pod \"konnectivity-agent-vcsls\" (UID: \"16eed615-a448-4416-b06e-1c33d1fffb6d\") " pod="kube-system/konnectivity-agent-vcsls" Nov 02 01:14:56.509346 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.509272 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-etc-openvswitch\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:56.509346 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.509294 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-etc-tuned\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:56.509774 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.509365 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6a02556d-4538-478d-84cf-a4d104565c84-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-snk78\" (UID: \"6a02556d-4538-478d-84cf-a4d104565c84\") " pod="openshift-multus/multus-additional-cni-plugins-snk78" Nov 02 01:14:56.509774 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.509400 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-tmp\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:56.509774 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.509417 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6a02556d-4538-478d-84cf-a4d104565c84-cni-binary-copy\") pod \"multus-additional-cni-plugins-snk78\" (UID: \"6a02556d-4538-478d-84cf-a4d104565c84\") " pod="openshift-multus/multus-additional-cni-plugins-snk78" Nov 02 01:14:56.509774 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.509436 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074-kubelet-config\") pod \"global-pull-secret-syncer-kl8v7\" (UID: \"9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074\") " pod="kube-system/global-pull-secret-syncer-kl8v7" Nov 02 01:14:56.509774 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.509452 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-tmp\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:56.509774 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.509395 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-etc-tuned\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:56.509774 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.509484 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41f79a83-9886-4c59-a904-328463dfe674-host\") pod \"node-ca-bgczm\" (UID: \"41f79a83-9886-4c59-a904-328463dfe674\") " pod="openshift-image-registry/node-ca-bgczm" Nov 02 01:14:56.509774 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.509471 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/16eed615-a448-4416-b06e-1c33d1fffb6d-agent-certs\") pod \"konnectivity-agent-vcsls\" (UID: \"16eed615-a448-4416-b06e-1c33d1fffb6d\") " pod="kube-system/konnectivity-agent-vcsls" Nov 02 01:14:56.509774 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.509535 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4a12c42f-81b2-4fde-bc01-8abe3e640957-cni-binary-copy\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:56.509774 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.509567 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c7b5fa9-d0b9-4147-bc52-bdd444a2c081-metrics-certs\") pod \"network-metrics-daemon-nkqjm\" (UID: \"4c7b5fa9-d0b9-4147-bc52-bdd444a2c081\") " pod="openshift-multus/network-metrics-daemon-nkqjm" Nov 02 01:14:56.509774 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.509591 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/23bf37a9-9b32-47f9-a4e0-364a7273c02f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gs62j\" (UID: \"23bf37a9-9b32-47f9-a4e0-364a7273c02f\") " pod="openshift-insights/insights-runtime-extractor-gs62j" Nov 02 01:14:56.509774 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.509618 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-host-slash\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:56.509774 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.509686 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-run-openvswitch\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:56.509774 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.509719 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-ovnkube-script-lib\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:56.509774 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.509754 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4a12c42f-81b2-4fde-bc01-8abe3e640957-cni-binary-copy\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:56.509774 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.509781 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-etc-modprobe-d\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:56.510523 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.509804 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxrsj\" (UniqueName: \"kubernetes.io/projected/3917d88c-88f5-4c5c-b1f7-e0195a61f88f-kube-api-access-hxrsj\") pod \"network-check-target-fdptg\" (UID: \"3917d88c-88f5-4c5c-b1f7-e0195a61f88f\") " pod="openshift-network-diagnostics/network-check-target-fdptg" Nov 02 01:14:56.510523 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.509820 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/20bc48c0-b0ef-40df-a5c9-06c48c75fe20-registration-dir\") pod \"aws-ebs-csi-driver-node-hj5fn\" (UID: \"20bc48c0-b0ef-40df-a5c9-06c48c75fe20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj5fn" Nov 02 01:14:56.510523 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.509861 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jtbq\" (UniqueName: \"kubernetes.io/projected/20bc48c0-b0ef-40df-a5c9-06c48c75fe20-kube-api-access-9jtbq\") pod \"aws-ebs-csi-driver-node-hj5fn\" (UID: \"20bc48c0-b0ef-40df-a5c9-06c48c75fe20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj5fn" Nov 02 01:14:56.510523 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.509943 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-ovnkube-script-lib\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:56.510523 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.509965 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-hostroot\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:56.510523 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.509996 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntxrh\" (UniqueName: \"kubernetes.io/projected/23bf37a9-9b32-47f9-a4e0-364a7273c02f-kube-api-access-ntxrh\") pod \"insights-runtime-extractor-gs62j\" (UID: \"23bf37a9-9b32-47f9-a4e0-364a7273c02f\") " pod="openshift-insights/insights-runtime-extractor-gs62j" Nov 02 01:14:56.510523 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.510033 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77df90af-e163-47ea-83ad-99bc1258f3d5-tmp-dir\") pod \"node-resolver-qwzn4\" (UID: \"77df90af-e163-47ea-83ad-99bc1258f3d5\") " pod="openshift-dns/node-resolver-qwzn4" Nov 02 01:14:56.510523 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.510068 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-sys\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:56.510523 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.510095 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/00d17dbb-cb72-4413-9559-4487c313158c-metrics-tls\") pod \"dns-default-xcm6b\" (UID: \"00d17dbb-cb72-4413-9559-4487c313158c\") " pod="openshift-dns/dns-default-xcm6b" Nov 02 01:14:56.510523 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.510116 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9pbx\" (UniqueName: \"kubernetes.io/projected/00d17dbb-cb72-4413-9559-4487c313158c-kube-api-access-g9pbx\") pod \"dns-default-xcm6b\" (UID: \"00d17dbb-cb72-4413-9559-4487c313158c\") " pod="openshift-dns/dns-default-xcm6b" Nov 02 01:14:56.510523 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.510175 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-host-run-k8s-cni-cncf-io\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:56.510523 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.510214 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/16eed615-a448-4416-b06e-1c33d1fffb6d-konnectivity-ca\") pod \"konnectivity-agent-vcsls\" (UID: \"16eed615-a448-4416-b06e-1c33d1fffb6d\") " pod="kube-system/konnectivity-agent-vcsls" Nov 02 01:14:56.510523 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.510213 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77df90af-e163-47ea-83ad-99bc1258f3d5-tmp-dir\") pod \"node-resolver-qwzn4\" (UID: \"77df90af-e163-47ea-83ad-99bc1258f3d5\") " pod="openshift-dns/node-resolver-qwzn4" Nov 02 01:14:56.510523 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.510241 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-var-lib-openvswitch\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:56.510523 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.510300 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s78hk\" (UniqueName: \"kubernetes.io/projected/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-kube-api-access-s78hk\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:56.510523 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.510334 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/41f79a83-9886-4c59-a904-328463dfe674-serviceca\") pod \"node-ca-bgczm\" (UID: \"41f79a83-9886-4c59-a904-328463dfe674\") " pod="openshift-image-registry/node-ca-bgczm" Nov 02 01:14:56.510523 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.510359 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/16eed615-a448-4416-b06e-1c33d1fffb6d-konnectivity-ca\") pod \"konnectivity-agent-vcsls\" (UID: \"16eed615-a448-4416-b06e-1c33d1fffb6d\") " pod="kube-system/konnectivity-agent-vcsls" Nov 02 01:14:56.510523 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.510361 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/20bc48c0-b0ef-40df-a5c9-06c48c75fe20-device-dir\") pod \"aws-ebs-csi-driver-node-hj5fn\" (UID: \"20bc48c0-b0ef-40df-a5c9-06c48c75fe20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj5fn" Nov 02 01:14:56.510523 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.510428 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-host-run-netns\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:56.510523 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.510453 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-host\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:56.510523 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.510508 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074-original-pull-secret\") pod \"global-pull-secret-syncer-kl8v7\" (UID: \"9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074\") " pod="kube-system/global-pull-secret-syncer-kl8v7" Nov 02 01:14:56.511266 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.510543 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/20bc48c0-b0ef-40df-a5c9-06c48c75fe20-socket-dir\") pod \"aws-ebs-csi-driver-node-hj5fn\" (UID: \"20bc48c0-b0ef-40df-a5c9-06c48c75fe20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj5fn" Nov 02 01:14:56.511266 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.510556 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/41f79a83-9886-4c59-a904-328463dfe674-serviceca\") pod \"node-ca-bgczm\" (UID: \"41f79a83-9886-4c59-a904-328463dfe674\") " pod="openshift-image-registry/node-ca-bgczm" Nov 02 01:14:56.511266 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.510573 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-cnibin\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:56.511266 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.510596 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6a02556d-4538-478d-84cf-a4d104565c84-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-snk78\" (UID: \"6a02556d-4538-478d-84cf-a4d104565c84\") " pod="openshift-multus/multus-additional-cni-plugins-snk78" Nov 02 01:14:56.511266 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.510620 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/23bf37a9-9b32-47f9-a4e0-364a7273c02f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gs62j\" (UID: \"23bf37a9-9b32-47f9-a4e0-364a7273c02f\") " pod="openshift-insights/insights-runtime-extractor-gs62j" Nov 02 01:14:56.511266 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.510672 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-host-kubelet\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:56.511266 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.510702 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-etc-sysconfig\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:56.511266 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.510729 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-var-lib-kubelet\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:56.511266 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.510752 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6a02556d-4538-478d-84cf-a4d104565c84-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-snk78\" (UID: \"6a02556d-4538-478d-84cf-a4d104565c84\") " pod="openshift-multus/multus-additional-cni-plugins-snk78" Nov 02 01:14:56.511266 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.510755 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074-original-pull-secret\") pod \"global-pull-secret-syncer-kl8v7\" (UID: \"9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074\") " pod="kube-system/global-pull-secret-syncer-kl8v7" Nov 02 01:14:56.511266 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.510756 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00d17dbb-cb72-4413-9559-4487c313158c-tmp-dir\") pod \"dns-default-xcm6b\" (UID: \"00d17dbb-cb72-4413-9559-4487c313158c\") " pod="openshift-dns/dns-default-xcm6b" Nov 02 01:14:56.511266 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.510788 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/20bc48c0-b0ef-40df-a5c9-06c48c75fe20-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hj5fn\" (UID: \"20bc48c0-b0ef-40df-a5c9-06c48c75fe20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj5fn" Nov 02 01:14:56.511266 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.510831 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-host-var-lib-cni-bin\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:56.511266 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.510836 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00d17dbb-cb72-4413-9559-4487c313158c-tmp-dir\") pod \"dns-default-xcm6b\" (UID: \"00d17dbb-cb72-4413-9559-4487c313158c\") " pod="openshift-dns/dns-default-xcm6b" Nov 02 01:14:56.511266 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.510859 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b98jt\" (UniqueName: \"kubernetes.io/projected/4c7b5fa9-d0b9-4147-bc52-bdd444a2c081-kube-api-access-b98jt\") pod \"network-metrics-daemon-nkqjm\" (UID: \"4c7b5fa9-d0b9-4147-bc52-bdd444a2c081\") " pod="openshift-multus/network-metrics-daemon-nkqjm" Nov 02 01:14:56.511266 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.510875 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-host-cni-bin\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:56.511266 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.510891 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-ovn-node-metrics-cert\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:56.511266 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.510905 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-etc-kubernetes\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:56.511266 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.510927 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-etc-kubernetes\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:56.511266 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.510942 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-log-socket\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:56.511266 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.510986 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-host-run-ovn-kubernetes\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:56.511266 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.511027 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/23bf37a9-9b32-47f9-a4e0-364a7273c02f-data-volume\") pod \"insights-runtime-extractor-gs62j\" (UID: \"23bf37a9-9b32-47f9-a4e0-364a7273c02f\") " pod="openshift-insights/insights-runtime-extractor-gs62j" Nov 02 01:14:56.511266 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.511067 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/497c02e6-eb5a-49ed-a911-97384fb409cf-iptables-alerter-script\") pod \"iptables-alerter-phtn9\" (UID: \"497c02e6-eb5a-49ed-a911-97384fb409cf\") " pod="openshift-network-operator/iptables-alerter-phtn9" Nov 02 01:14:56.511266 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.511095 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-etc-systemd\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:56.511266 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.511121 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-ovn-node-metrics-cert\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:56.511266 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.511131 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a75a9f45-7678-4014-92d9-93d3f5b260ba-cert\") pod \"ingress-canary-pn4kl\" (UID: \"a75a9f45-7678-4014-92d9-93d3f5b260ba\") " pod="openshift-ingress-canary/ingress-canary-pn4kl" Nov 02 01:14:56.511266 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.511170 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-host-var-lib-cni-multus\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:56.511266 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.511189 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/77df90af-e163-47ea-83ad-99bc1258f3d5-hosts-file\") pod \"node-resolver-qwzn4\" (UID: \"77df90af-e163-47ea-83ad-99bc1258f3d5\") " pod="openshift-dns/node-resolver-qwzn4" Nov 02 01:14:56.511266 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.511198 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/23bf37a9-9b32-47f9-a4e0-364a7273c02f-data-volume\") pod \"insights-runtime-extractor-gs62j\" (UID: \"23bf37a9-9b32-47f9-a4e0-364a7273c02f\") " pod="openshift-insights/insights-runtime-extractor-gs62j" Nov 02 01:14:56.511266 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.511207 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p26j\" (UniqueName: \"kubernetes.io/projected/497c02e6-eb5a-49ed-a911-97384fb409cf-kube-api-access-4p26j\") pod \"iptables-alerter-phtn9\" (UID: \"497c02e6-eb5a-49ed-a911-97384fb409cf\") " pod="openshift-network-operator/iptables-alerter-phtn9" Nov 02 01:14:56.511266 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.511245 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khqc7\" (UniqueName: \"kubernetes.io/projected/41f79a83-9886-4c59-a904-328463dfe674-kube-api-access-khqc7\") pod \"node-ca-bgczm\" (UID: \"41f79a83-9886-4c59-a904-328463dfe674\") " pod="openshift-image-registry/node-ca-bgczm" Nov 02 01:14:56.511266 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.511280 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/497c02e6-eb5a-49ed-a911-97384fb409cf-iptables-alerter-script\") pod \"iptables-alerter-phtn9\" (UID: \"497c02e6-eb5a-49ed-a911-97384fb409cf\") " pod="openshift-network-operator/iptables-alerter-phtn9" Nov 02 01:14:56.512279 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.511335 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a75a9f45-7678-4014-92d9-93d3f5b260ba-cert\") pod \"ingress-canary-pn4kl\" (UID: \"a75a9f45-7678-4014-92d9-93d3f5b260ba\") " pod="openshift-ingress-canary/ingress-canary-pn4kl" Nov 02 01:14:56.512279 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.511352 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-system-cni-dir\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:56.512279 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.511417 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-host-cni-netd\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:56.512279 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.511448 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:56.512279 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.511473 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-987n6\" (UniqueName: \"kubernetes.io/projected/a75a9f45-7678-4014-92d9-93d3f5b260ba-kube-api-access-987n6\") pod \"ingress-canary-pn4kl\" (UID: \"a75a9f45-7678-4014-92d9-93d3f5b260ba\") " pod="openshift-ingress-canary/ingress-canary-pn4kl" Nov 02 01:14:56.512279 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.511584 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/20bc48c0-b0ef-40df-a5c9-06c48c75fe20-etc-selinux\") pod \"aws-ebs-csi-driver-node-hj5fn\" (UID: \"20bc48c0-b0ef-40df-a5c9-06c48c75fe20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj5fn" Nov 02 01:14:56.512279 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.511629 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-run-ovn\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:56.512279 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.511646 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-node-log\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:56.512279 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.511660 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-etc-sysctl-d\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:56.519345 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.519327 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlbbj\" (UniqueName: \"kubernetes.io/projected/6a02556d-4538-478d-84cf-a4d104565c84-kube-api-access-nlbbj\") pod \"multus-additional-cni-plugins-snk78\" (UID: \"6a02556d-4538-478d-84cf-a4d104565c84\") " pod="openshift-multus/multus-additional-cni-plugins-snk78" Nov 02 01:14:56.519489 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.519473 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8tts\" (UniqueName: \"kubernetes.io/projected/77df90af-e163-47ea-83ad-99bc1258f3d5-kube-api-access-c8tts\") pod \"node-resolver-qwzn4\" (UID: \"77df90af-e163-47ea-83ad-99bc1258f3d5\") " pod="openshift-dns/node-resolver-qwzn4" Nov 02 01:14:56.519557 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.519538 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd76d\" (UniqueName: \"kubernetes.io/projected/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-kube-api-access-pd76d\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:56.519590 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.519538 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs7p2\" (UniqueName: \"kubernetes.io/projected/4a12c42f-81b2-4fde-bc01-8abe3e640957-kube-api-access-bs7p2\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:56.520643 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.520619 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9pbx\" (UniqueName: \"kubernetes.io/projected/00d17dbb-cb72-4413-9559-4487c313158c-kube-api-access-g9pbx\") pod \"dns-default-xcm6b\" (UID: \"00d17dbb-cb72-4413-9559-4487c313158c\") " pod="openshift-dns/dns-default-xcm6b" Nov 02 01:14:56.520733 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.520659 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jtbq\" (UniqueName: \"kubernetes.io/projected/20bc48c0-b0ef-40df-a5c9-06c48c75fe20-kube-api-access-9jtbq\") pod \"aws-ebs-csi-driver-node-hj5fn\" (UID: \"20bc48c0-b0ef-40df-a5c9-06c48c75fe20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj5fn" Nov 02 01:14:56.520807 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.520793 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxrsj\" (UniqueName: \"kubernetes.io/projected/3917d88c-88f5-4c5c-b1f7-e0195a61f88f-kube-api-access-hxrsj\") pod \"network-check-target-fdptg\" (UID: \"3917d88c-88f5-4c5c-b1f7-e0195a61f88f\") " pod="openshift-network-diagnostics/network-check-target-fdptg" Nov 02 01:14:56.520930 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.520915 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p26j\" (UniqueName: \"kubernetes.io/projected/497c02e6-eb5a-49ed-a911-97384fb409cf-kube-api-access-4p26j\") pod \"iptables-alerter-phtn9\" (UID: \"497c02e6-eb5a-49ed-a911-97384fb409cf\") " pod="openshift-network-operator/iptables-alerter-phtn9" Nov 02 01:14:56.521927 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.521911 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s78hk\" (UniqueName: \"kubernetes.io/projected/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-kube-api-access-s78hk\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:56.521962 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.521927 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khqc7\" (UniqueName: \"kubernetes.io/projected/41f79a83-9886-4c59-a904-328463dfe674-kube-api-access-khqc7\") pod \"node-ca-bgczm\" (UID: \"41f79a83-9886-4c59-a904-328463dfe674\") " pod="openshift-image-registry/node-ca-bgczm" Nov 02 01:14:56.524297 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.524275 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntxrh\" (UniqueName: \"kubernetes.io/projected/23bf37a9-9b32-47f9-a4e0-364a7273c02f-kube-api-access-ntxrh\") pod \"insights-runtime-extractor-gs62j\" (UID: \"23bf37a9-9b32-47f9-a4e0-364a7273c02f\") " pod="openshift-insights/insights-runtime-extractor-gs62j" Nov 02 01:14:56.524393 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.524305 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b98jt\" (UniqueName: \"kubernetes.io/projected/4c7b5fa9-d0b9-4147-bc52-bdd444a2c081-kube-api-access-b98jt\") pod \"network-metrics-daemon-nkqjm\" (UID: \"4c7b5fa9-d0b9-4147-bc52-bdd444a2c081\") " pod="openshift-multus/network-metrics-daemon-nkqjm" Nov 02 01:14:56.524485 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.524466 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-987n6\" (UniqueName: \"kubernetes.io/projected/a75a9f45-7678-4014-92d9-93d3f5b260ba-kube-api-access-987n6\") pod \"ingress-canary-pn4kl\" (UID: \"a75a9f45-7678-4014-92d9-93d3f5b260ba\") " pod="openshift-ingress-canary/ingress-canary-pn4kl" Nov 02 01:14:56.560386 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.560355 4427 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-130.us-east-2.compute.internal" Nov 02 01:14:56.560530 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.560485 4427 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-130.us-east-2.compute.internal" Nov 02 01:14:56.571003 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.570978 4427 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 01:14:56.571003 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.570991 4427 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 01:14:56.571115 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:14:56.571024 4427 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-130.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-130.us-east-2.compute.internal" Nov 02 01:14:56.571115 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:14:56.571027 4427 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-130.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-130.us-east-2.compute.internal" Nov 02 01:14:56.589258 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.589235 4427 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 01:14:56.612057 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.612026 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-host-run-k8s-cni-cncf-io\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:56.612057 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.612057 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-var-lib-openvswitch\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:56.612227 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.612079 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/20bc48c0-b0ef-40df-a5c9-06c48c75fe20-device-dir\") pod \"aws-ebs-csi-driver-node-hj5fn\" (UID: \"20bc48c0-b0ef-40df-a5c9-06c48c75fe20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj5fn" Nov 02 01:14:56.612227 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.612097 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-host-run-netns\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:56.612227 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.612112 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-host\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:56.612227 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.612134 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-host-run-k8s-cni-cncf-io\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:56.612227 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.612136 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/20bc48c0-b0ef-40df-a5c9-06c48c75fe20-socket-dir\") pod \"aws-ebs-csi-driver-node-hj5fn\" (UID: \"20bc48c0-b0ef-40df-a5c9-06c48c75fe20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj5fn" Nov 02 01:14:56.612227 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.612161 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-var-lib-openvswitch\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:56.612227 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.612189 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/20bc48c0-b0ef-40df-a5c9-06c48c75fe20-device-dir\") pod \"aws-ebs-csi-driver-node-hj5fn\" (UID: \"20bc48c0-b0ef-40df-a5c9-06c48c75fe20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj5fn" Nov 02 01:14:56.612227 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.612186 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-host-run-netns\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:56.612597 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.612217 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/20bc48c0-b0ef-40df-a5c9-06c48c75fe20-socket-dir\") pod \"aws-ebs-csi-driver-node-hj5fn\" (UID: \"20bc48c0-b0ef-40df-a5c9-06c48c75fe20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj5fn" Nov 02 01:14:56.612597 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.612237 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-cnibin\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:56.612597 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.612246 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-host\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:56.612597 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.612268 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/23bf37a9-9b32-47f9-a4e0-364a7273c02f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gs62j\" (UID: \"23bf37a9-9b32-47f9-a4e0-364a7273c02f\") " pod="openshift-insights/insights-runtime-extractor-gs62j" Nov 02 01:14:56.612597 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.612296 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-host-kubelet\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:56.612597 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.612312 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-cnibin\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:56.612597 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.612321 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-etc-sysconfig\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:56.612597 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.612355 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-var-lib-kubelet\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:56.612597 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.612361 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-host-kubelet\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:56.612597 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.612367 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-etc-sysconfig\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:56.612597 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.612400 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/20bc48c0-b0ef-40df-a5c9-06c48c75fe20-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hj5fn\" (UID: \"20bc48c0-b0ef-40df-a5c9-06c48c75fe20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj5fn" Nov 02 01:14:56.612597 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.612428 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-var-lib-kubelet\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:56.612597 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.612440 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-host-var-lib-cni-bin\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:56.612597 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.612474 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-host-cni-bin\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:56.612597 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.612480 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/20bc48c0-b0ef-40df-a5c9-06c48c75fe20-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hj5fn\" (UID: \"20bc48c0-b0ef-40df-a5c9-06c48c75fe20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj5fn" Nov 02 01:14:56.612597 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.612500 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-etc-kubernetes\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:56.612597 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.612512 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-host-var-lib-cni-bin\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:56.612597 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.612526 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dg8x\" (UniqueName: \"kubernetes.io/projected/212ace3e-7830-4810-938d-22fd7f251b25-kube-api-access-7dg8x\") pod \"migrator-66df5d496b-ctstx\" (UID: \"212ace3e-7830-4810-938d-22fd7f251b25\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ctstx" Nov 02 01:14:56.612597 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.612529 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-host-cni-bin\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:56.612597 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.612550 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-etc-kubernetes\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:56.612597 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.612569 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-etc-kubernetes\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:56.612597 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.612567 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-log-socket\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:56.613484 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.612618 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-etc-kubernetes\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:56.613484 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.612639 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-log-socket\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:56.613484 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.612648 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-host-run-ovn-kubernetes\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:56.613484 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.612677 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-etc-systemd\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:56.613484 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.612697 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-host-run-ovn-kubernetes\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:56.613484 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.612703 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-host-var-lib-cni-multus\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:56.613484 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.612758 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-etc-systemd\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:56.613484 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.612765 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/77df90af-e163-47ea-83ad-99bc1258f3d5-hosts-file\") pod \"node-resolver-qwzn4\" (UID: \"77df90af-e163-47ea-83ad-99bc1258f3d5\") " pod="openshift-dns/node-resolver-qwzn4" Nov 02 01:14:56.613484 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.612818 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-host-var-lib-cni-multus\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:56.613484 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.612803 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-system-cni-dir\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:56.613484 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.612859 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/23bf37a9-9b32-47f9-a4e0-364a7273c02f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gs62j\" (UID: \"23bf37a9-9b32-47f9-a4e0-364a7273c02f\") " pod="openshift-insights/insights-runtime-extractor-gs62j" Nov 02 01:14:56.613484 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.612892 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-host-cni-netd\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:56.613484 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.612886 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/77df90af-e163-47ea-83ad-99bc1258f3d5-hosts-file\") pod \"node-resolver-qwzn4\" (UID: \"77df90af-e163-47ea-83ad-99bc1258f3d5\") " pod="openshift-dns/node-resolver-qwzn4" Nov 02 01:14:56.613484 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.612921 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-system-cni-dir\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:56.613484 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.612932 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:56.613484 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.612963 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-host-cni-netd\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:56.613484 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.612980 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/20bc48c0-b0ef-40df-a5c9-06c48c75fe20-etc-selinux\") pod \"aws-ebs-csi-driver-node-hj5fn\" (UID: \"20bc48c0-b0ef-40df-a5c9-06c48c75fe20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj5fn" Nov 02 01:14:56.613484 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.613015 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-run-ovn\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:56.613484 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.613032 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:56.613484 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.613037 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-node-log\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:56.613484 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.613053 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/20bc48c0-b0ef-40df-a5c9-06c48c75fe20-etc-selinux\") pod \"aws-ebs-csi-driver-node-hj5fn\" (UID: \"20bc48c0-b0ef-40df-a5c9-06c48c75fe20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj5fn" Nov 02 01:14:56.613484 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.613063 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-run-ovn\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:56.613484 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.613074 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-node-log\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:56.613484 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.613082 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-etc-sysctl-d\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:56.613484 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.613104 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-lib-modules\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:56.613484 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.613121 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074-dbus\") pod \"global-pull-secret-syncer-kl8v7\" (UID: \"9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074\") " pod="kube-system/global-pull-secret-syncer-kl8v7" Nov 02 01:14:56.613484 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.613137 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6a02556d-4538-478d-84cf-a4d104565c84-cnibin\") pod \"multus-additional-cni-plugins-snk78\" (UID: \"6a02556d-4538-478d-84cf-a4d104565c84\") " pod="openshift-multus/multus-additional-cni-plugins-snk78" Nov 02 01:14:56.613484 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.613163 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-systemd-units\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:56.613484 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.613185 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-run\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:56.613484 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.613207 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074-dbus\") pod \"global-pull-secret-syncer-kl8v7\" (UID: \"9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074\") " pod="kube-system/global-pull-secret-syncer-kl8v7" Nov 02 01:14:56.613484 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.613211 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/20bc48c0-b0ef-40df-a5c9-06c48c75fe20-sys-fs\") pod \"aws-ebs-csi-driver-node-hj5fn\" (UID: \"20bc48c0-b0ef-40df-a5c9-06c48c75fe20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj5fn" Nov 02 01:14:56.613484 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.613220 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-systemd-units\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:56.613484 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.613236 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-run\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:56.613484 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.613253 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/20bc48c0-b0ef-40df-a5c9-06c48c75fe20-sys-fs\") pod \"aws-ebs-csi-driver-node-hj5fn\" (UID: \"20bc48c0-b0ef-40df-a5c9-06c48c75fe20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj5fn" Nov 02 01:14:56.613484 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.613242 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-etc-sysctl-d\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:56.613484 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.613241 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-host-run-multus-certs\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:56.613484 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.613243 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-lib-modules\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:56.613484 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.613296 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-host-run-multus-certs\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:56.613484 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.613337 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a02556d-4538-478d-84cf-a4d104565c84-system-cni-dir\") pod \"multus-additional-cni-plugins-snk78\" (UID: \"6a02556d-4538-478d-84cf-a4d104565c84\") " pod="openshift-multus/multus-additional-cni-plugins-snk78" Nov 02 01:14:56.613484 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.613344 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6a02556d-4538-478d-84cf-a4d104565c84-cnibin\") pod \"multus-additional-cni-plugins-snk78\" (UID: \"6a02556d-4538-478d-84cf-a4d104565c84\") " pod="openshift-multus/multus-additional-cni-plugins-snk78" Nov 02 01:14:56.613484 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.613353 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6a02556d-4538-478d-84cf-a4d104565c84-tuning-conf-dir\") pod \"multus-additional-cni-plugins-snk78\" (UID: \"6a02556d-4538-478d-84cf-a4d104565c84\") " pod="openshift-multus/multus-additional-cni-plugins-snk78" Nov 02 01:14:56.613484 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.613412 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-host-run-netns\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:56.613484 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.613435 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6a02556d-4538-478d-84cf-a4d104565c84-tuning-conf-dir\") pod \"multus-additional-cni-plugins-snk78\" (UID: \"6a02556d-4538-478d-84cf-a4d104565c84\") " pod="openshift-multus/multus-additional-cni-plugins-snk78" Nov 02 01:14:56.613484 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.613444 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-host-var-lib-kubelet\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:56.613484 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.613413 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a02556d-4538-478d-84cf-a4d104565c84-system-cni-dir\") pod \"multus-additional-cni-plugins-snk78\" (UID: \"6a02556d-4538-478d-84cf-a4d104565c84\") " pod="openshift-multus/multus-additional-cni-plugins-snk78" Nov 02 01:14:56.613484 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.613467 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-multus-conf-dir\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:56.615127 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.613527 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/23bf37a9-9b32-47f9-a4e0-364a7273c02f-crio-socket\") pod \"insights-runtime-extractor-gs62j\" (UID: \"23bf37a9-9b32-47f9-a4e0-364a7273c02f\") " pod="openshift-insights/insights-runtime-extractor-gs62j" Nov 02 01:14:56.615127 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.613550 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-host-var-lib-kubelet\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:56.615127 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.613529 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-multus-conf-dir\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:56.615127 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.613578 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-multus-socket-dir-parent\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:56.615127 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.613644 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/23bf37a9-9b32-47f9-a4e0-364a7273c02f-crio-socket\") pod \"insights-runtime-extractor-gs62j\" (UID: \"23bf37a9-9b32-47f9-a4e0-364a7273c02f\") " pod="openshift-insights/insights-runtime-extractor-gs62j" Nov 02 01:14:56.615127 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.613589 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-host-run-netns\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:56.615127 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.613663 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-run-systemd\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:56.615127 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.613672 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-multus-socket-dir-parent\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:56.615127 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.613714 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-run-systemd\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:56.615127 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.613831 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/497c02e6-eb5a-49ed-a911-97384fb409cf-host-slash\") pod \"iptables-alerter-phtn9\" (UID: \"497c02e6-eb5a-49ed-a911-97384fb409cf\") " pod="openshift-network-operator/iptables-alerter-phtn9" Nov 02 01:14:56.615127 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.613708 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/497c02e6-eb5a-49ed-a911-97384fb409cf-host-slash\") pod \"iptables-alerter-phtn9\" (UID: \"497c02e6-eb5a-49ed-a911-97384fb409cf\") " pod="openshift-network-operator/iptables-alerter-phtn9" Nov 02 01:14:56.615127 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.614162 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-etc-sysctl-conf\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:56.615127 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.614181 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-multus-cni-dir\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:56.615127 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.614198 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-etc-openvswitch\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:56.615127 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.614213 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074-kubelet-config\") pod \"global-pull-secret-syncer-kl8v7\" (UID: \"9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074\") " pod="kube-system/global-pull-secret-syncer-kl8v7" Nov 02 01:14:56.615127 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.614235 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41f79a83-9886-4c59-a904-328463dfe674-host\") pod \"node-ca-bgczm\" (UID: \"41f79a83-9886-4c59-a904-328463dfe674\") " pod="openshift-image-registry/node-ca-bgczm" Nov 02 01:14:56.615127 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.614280 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41f79a83-9886-4c59-a904-328463dfe674-host\") pod \"node-ca-bgczm\" (UID: \"41f79a83-9886-4c59-a904-328463dfe674\") " pod="openshift-image-registry/node-ca-bgczm" Nov 02 01:14:56.615127 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.614281 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-etc-openvswitch\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:56.615127 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.614292 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074-kubelet-config\") pod \"global-pull-secret-syncer-kl8v7\" (UID: \"9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074\") " pod="kube-system/global-pull-secret-syncer-kl8v7" Nov 02 01:14:56.615127 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.614279 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-multus-cni-dir\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:56.615127 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.614305 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c7b5fa9-d0b9-4147-bc52-bdd444a2c081-metrics-certs\") pod \"network-metrics-daemon-nkqjm\" (UID: \"4c7b5fa9-d0b9-4147-bc52-bdd444a2c081\") " pod="openshift-multus/network-metrics-daemon-nkqjm" Nov 02 01:14:56.615127 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.614335 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/23bf37a9-9b32-47f9-a4e0-364a7273c02f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gs62j\" (UID: \"23bf37a9-9b32-47f9-a4e0-364a7273c02f\") " pod="openshift-insights/insights-runtime-extractor-gs62j" Nov 02 01:14:56.615127 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.614359 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-etc-sysctl-conf\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:56.615127 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.614360 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-host-slash\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:56.615127 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.614436 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-host-slash\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:56.615127 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:14:56.614392 4427 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 01:14:56.615127 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.614433 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-run-openvswitch\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:56.615127 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.614484 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7-run-openvswitch\") pod \"ovnkube-node-z8bzg\" (UID: \"84e88ab0-7616-4917-ad8f-ae3c8b5a0ee7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:14:56.615127 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.614509 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-etc-modprobe-d\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:56.615127 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:14:56.614440 4427 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 01:14:56.615127 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.614562 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/20bc48c0-b0ef-40df-a5c9-06c48c75fe20-registration-dir\") pod \"aws-ebs-csi-driver-node-hj5fn\" (UID: \"20bc48c0-b0ef-40df-a5c9-06c48c75fe20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj5fn" Nov 02 01:14:56.615127 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:14:56.614603 4427 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c7b5fa9-d0b9-4147-bc52-bdd444a2c081-metrics-certs podName:4c7b5fa9-d0b9-4147-bc52-bdd444a2c081 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:57.114579551 +0000 UTC m=+1.735584757 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4c7b5fa9-d0b9-4147-bc52-bdd444a2c081-metrics-certs") pod "network-metrics-daemon-nkqjm" (UID: "4c7b5fa9-d0b9-4147-bc52-bdd444a2c081") : secret "metrics-daemon-secret" not found Nov 02 01:14:56.615127 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.614614 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-etc-modprobe-d\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:56.615127 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.614623 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-hostroot\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:56.615127 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.614625 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/20bc48c0-b0ef-40df-a5c9-06c48c75fe20-registration-dir\") pod \"aws-ebs-csi-driver-node-hj5fn\" (UID: \"20bc48c0-b0ef-40df-a5c9-06c48c75fe20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj5fn" Nov 02 01:14:56.615127 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.614697 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-sys\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:56.615127 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.614721 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4a12c42f-81b2-4fde-bc01-8abe3e640957-hostroot\") pod \"multus-27jtx\" (UID: \"4a12c42f-81b2-4fde-bc01-8abe3e640957\") " pod="openshift-multus/multus-27jtx" Nov 02 01:14:56.615127 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:14:56.614726 4427 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23bf37a9-9b32-47f9-a4e0-364a7273c02f-insights-runtime-extractor-tls podName:23bf37a9-9b32-47f9-a4e0-364a7273c02f nodeName:}" failed. No retries permitted until 2026-11-02 01:14:57.114714041 +0000 UTC m=+1.735719238 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/23bf37a9-9b32-47f9-a4e0-364a7273c02f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gs62j" (UID: "23bf37a9-9b32-47f9-a4e0-364a7273c02f") : secret "insights-runtime-extractor-tls" not found Nov 02 01:14:56.615127 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.614753 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8cd0f6e2-e491-4b6c-943a-c1115300dcaf-sys\") pod \"tuned-pqmz2\" (UID: \"8cd0f6e2-e491-4b6c-943a-c1115300dcaf\") " pod="openshift-cluster-node-tuning-operator/tuned-pqmz2" Nov 02 01:14:56.615127 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.614757 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/00d17dbb-cb72-4413-9559-4487c313158c-metrics-tls\") pod \"dns-default-xcm6b\" (UID: \"00d17dbb-cb72-4413-9559-4487c313158c\") " pod="openshift-dns/dns-default-xcm6b" Nov 02 01:14:56.615127 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:14:56.614820 4427 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 01:14:56.615127 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:14:56.614868 4427 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00d17dbb-cb72-4413-9559-4487c313158c-metrics-tls podName:00d17dbb-cb72-4413-9559-4487c313158c nodeName:}" failed. No retries permitted until 2026-11-02 01:14:57.11484663 +0000 UTC m=+1.735851819 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/00d17dbb-cb72-4413-9559-4487c313158c-metrics-tls") pod "dns-default-xcm6b" (UID: "00d17dbb-cb72-4413-9559-4487c313158c") : secret "dns-default-metrics-tls" not found Nov 02 01:14:56.622149 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:56.622125 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dg8x\" (UniqueName: \"kubernetes.io/projected/212ace3e-7830-4810-938d-22fd7f251b25-kube-api-access-7dg8x\") pod \"migrator-66df5d496b-ctstx\" (UID: \"212ace3e-7830-4810-938d-22fd7f251b25\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ctstx" Nov 02 01:14:57.118093 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:57.118047 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c7b5fa9-d0b9-4147-bc52-bdd444a2c081-metrics-certs\") pod \"network-metrics-daemon-nkqjm\" (UID: \"4c7b5fa9-d0b9-4147-bc52-bdd444a2c081\") " pod="openshift-multus/network-metrics-daemon-nkqjm" Nov 02 01:14:57.118093 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:57.118094 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/23bf37a9-9b32-47f9-a4e0-364a7273c02f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gs62j\" (UID: \"23bf37a9-9b32-47f9-a4e0-364a7273c02f\") " pod="openshift-insights/insights-runtime-extractor-gs62j" Nov 02 01:14:57.118300 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:57.118116 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/00d17dbb-cb72-4413-9559-4487c313158c-metrics-tls\") pod \"dns-default-xcm6b\" (UID: \"00d17dbb-cb72-4413-9559-4487c313158c\") " pod="openshift-dns/dns-default-xcm6b" Nov 02 01:14:57.118300 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:14:57.118223 4427 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 01:14:57.118300 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:14:57.118237 4427 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 01:14:57.118300 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:14:57.118260 4427 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 01:14:57.118300 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:14:57.118300 4427 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c7b5fa9-d0b9-4147-bc52-bdd444a2c081-metrics-certs podName:4c7b5fa9-d0b9-4147-bc52-bdd444a2c081 nodeName:}" failed. No retries permitted until 2026-11-02 01:14:58.11828054 +0000 UTC m=+2.739285747 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4c7b5fa9-d0b9-4147-bc52-bdd444a2c081-metrics-certs") pod "network-metrics-daemon-nkqjm" (UID: "4c7b5fa9-d0b9-4147-bc52-bdd444a2c081") : secret "metrics-daemon-secret" not found Nov 02 01:14:57.118460 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:14:57.118319 4427 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23bf37a9-9b32-47f9-a4e0-364a7273c02f-insights-runtime-extractor-tls podName:23bf37a9-9b32-47f9-a4e0-364a7273c02f nodeName:}" failed. No retries permitted until 2026-11-02 01:14:58.118311159 +0000 UTC m=+2.739316348 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/23bf37a9-9b32-47f9-a4e0-364a7273c02f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gs62j" (UID: "23bf37a9-9b32-47f9-a4e0-364a7273c02f") : secret "insights-runtime-extractor-tls" not found Nov 02 01:14:57.118460 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:14:57.118345 4427 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00d17dbb-cb72-4413-9559-4487c313158c-metrics-tls podName:00d17dbb-cb72-4413-9559-4487c313158c nodeName:}" failed. No retries permitted until 2026-11-02 01:14:58.118335211 +0000 UTC m=+2.739340400 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/00d17dbb-cb72-4413-9559-4487c313158c-metrics-tls") pod "dns-default-xcm6b" (UID: "00d17dbb-cb72-4413-9559-4487c313158c") : secret "dns-default-metrics-tls" not found Nov 02 01:14:58.124430 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:58.124369 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c7b5fa9-d0b9-4147-bc52-bdd444a2c081-metrics-certs\") pod \"network-metrics-daemon-nkqjm\" (UID: \"4c7b5fa9-d0b9-4147-bc52-bdd444a2c081\") " pod="openshift-multus/network-metrics-daemon-nkqjm" Nov 02 01:14:58.124801 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:58.124441 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/23bf37a9-9b32-47f9-a4e0-364a7273c02f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gs62j\" (UID: \"23bf37a9-9b32-47f9-a4e0-364a7273c02f\") " pod="openshift-insights/insights-runtime-extractor-gs62j" Nov 02 01:14:58.124801 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:14:58.124468 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/00d17dbb-cb72-4413-9559-4487c313158c-metrics-tls\") pod \"dns-default-xcm6b\" (UID: \"00d17dbb-cb72-4413-9559-4487c313158c\") " pod="openshift-dns/dns-default-xcm6b" Nov 02 01:14:58.124801 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:14:58.124495 4427 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 01:14:58.124801 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:14:58.124560 4427 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c7b5fa9-d0b9-4147-bc52-bdd444a2c081-metrics-certs podName:4c7b5fa9-d0b9-4147-bc52-bdd444a2c081 nodeName:}" failed. No retries permitted until 2026-11-02 01:15:00.124546141 +0000 UTC m=+4.745551334 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4c7b5fa9-d0b9-4147-bc52-bdd444a2c081-metrics-certs") pod "network-metrics-daemon-nkqjm" (UID: "4c7b5fa9-d0b9-4147-bc52-bdd444a2c081") : secret "metrics-daemon-secret" not found Nov 02 01:14:58.124801 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:14:58.124585 4427 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 01:14:58.124801 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:14:58.124585 4427 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 01:14:58.124801 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:14:58.124641 4427 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23bf37a9-9b32-47f9-a4e0-364a7273c02f-insights-runtime-extractor-tls podName:23bf37a9-9b32-47f9-a4e0-364a7273c02f nodeName:}" failed. No retries permitted until 2026-11-02 01:15:00.124623589 +0000 UTC m=+4.745628781 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/23bf37a9-9b32-47f9-a4e0-364a7273c02f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gs62j" (UID: "23bf37a9-9b32-47f9-a4e0-364a7273c02f") : secret "insights-runtime-extractor-tls" not found Nov 02 01:14:58.124801 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:14:58.124661 4427 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00d17dbb-cb72-4413-9559-4487c313158c-metrics-tls podName:00d17dbb-cb72-4413-9559-4487c313158c nodeName:}" failed. No retries permitted until 2026-11-02 01:15:00.124650684 +0000 UTC m=+4.745655875 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/00d17dbb-cb72-4413-9559-4487c313158c-metrics-tls") pod "dns-default-xcm6b" (UID: "00d17dbb-cb72-4413-9559-4487c313158c") : secret "dns-default-metrics-tls" not found Nov 02 01:15:00.136478 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:00.136425 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c7b5fa9-d0b9-4147-bc52-bdd444a2c081-metrics-certs\") pod \"network-metrics-daemon-nkqjm\" (UID: \"4c7b5fa9-d0b9-4147-bc52-bdd444a2c081\") " pod="openshift-multus/network-metrics-daemon-nkqjm" Nov 02 01:15:00.136478 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:00.136476 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/23bf37a9-9b32-47f9-a4e0-364a7273c02f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gs62j\" (UID: \"23bf37a9-9b32-47f9-a4e0-364a7273c02f\") " pod="openshift-insights/insights-runtime-extractor-gs62j" Nov 02 01:15:00.137003 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:00.136499 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/00d17dbb-cb72-4413-9559-4487c313158c-metrics-tls\") pod \"dns-default-xcm6b\" (UID: \"00d17dbb-cb72-4413-9559-4487c313158c\") " pod="openshift-dns/dns-default-xcm6b" Nov 02 01:15:00.137003 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:15:00.136582 4427 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 01:15:00.137003 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:15:00.136606 4427 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 01:15:00.137003 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:15:00.136619 4427 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 01:15:00.137003 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:15:00.136642 4427 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c7b5fa9-d0b9-4147-bc52-bdd444a2c081-metrics-certs podName:4c7b5fa9-d0b9-4147-bc52-bdd444a2c081 nodeName:}" failed. No retries permitted until 2026-11-02 01:15:04.136629041 +0000 UTC m=+8.757634233 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4c7b5fa9-d0b9-4147-bc52-bdd444a2c081-metrics-certs") pod "network-metrics-daemon-nkqjm" (UID: "4c7b5fa9-d0b9-4147-bc52-bdd444a2c081") : secret "metrics-daemon-secret" not found Nov 02 01:15:00.137003 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:15:00.136658 4427 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00d17dbb-cb72-4413-9559-4487c313158c-metrics-tls podName:00d17dbb-cb72-4413-9559-4487c313158c nodeName:}" failed. No retries permitted until 2026-11-02 01:15:04.136647998 +0000 UTC m=+8.757653187 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/00d17dbb-cb72-4413-9559-4487c313158c-metrics-tls") pod "dns-default-xcm6b" (UID: "00d17dbb-cb72-4413-9559-4487c313158c") : secret "dns-default-metrics-tls" not found Nov 02 01:15:00.137003 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:15:00.136670 4427 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23bf37a9-9b32-47f9-a4e0-364a7273c02f-insights-runtime-extractor-tls podName:23bf37a9-9b32-47f9-a4e0-364a7273c02f nodeName:}" failed. No retries permitted until 2026-11-02 01:15:04.136663899 +0000 UTC m=+8.757669088 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/23bf37a9-9b32-47f9-a4e0-364a7273c02f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gs62j" (UID: "23bf37a9-9b32-47f9-a4e0-364a7273c02f") : secret "insights-runtime-extractor-tls" not found Nov 02 01:15:01.221030 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:01.221003 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:15:03.779542 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:03.779515 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qwzn4_77df90af-e163-47ea-83ad-99bc1258f3d5/dns-node-resolver/0.log" Nov 02 01:15:03.843809 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:03.843778 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-fdptg" Nov 02 01:15:04.167195 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:04.167152 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/00d17dbb-cb72-4413-9559-4487c313158c-metrics-tls\") pod \"dns-default-xcm6b\" (UID: \"00d17dbb-cb72-4413-9559-4487c313158c\") " pod="openshift-dns/dns-default-xcm6b" Nov 02 01:15:04.167457 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:04.167233 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c7b5fa9-d0b9-4147-bc52-bdd444a2c081-metrics-certs\") pod \"network-metrics-daemon-nkqjm\" (UID: \"4c7b5fa9-d0b9-4147-bc52-bdd444a2c081\") " pod="openshift-multus/network-metrics-daemon-nkqjm" Nov 02 01:15:04.167457 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:04.167261 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/23bf37a9-9b32-47f9-a4e0-364a7273c02f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gs62j\" (UID: \"23bf37a9-9b32-47f9-a4e0-364a7273c02f\") " pod="openshift-insights/insights-runtime-extractor-gs62j" Nov 02 01:15:04.167457 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:15:04.167362 4427 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 01:15:04.167457 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:15:04.167390 4427 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 01:15:04.167457 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:15:04.167458 4427 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c7b5fa9-d0b9-4147-bc52-bdd444a2c081-metrics-certs podName:4c7b5fa9-d0b9-4147-bc52-bdd444a2c081 nodeName:}" failed. No retries permitted until 2026-11-02 01:15:12.167444671 +0000 UTC m=+16.788449891 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4c7b5fa9-d0b9-4147-bc52-bdd444a2c081-metrics-certs") pod "network-metrics-daemon-nkqjm" (UID: "4c7b5fa9-d0b9-4147-bc52-bdd444a2c081") : secret "metrics-daemon-secret" not found Nov 02 01:15:04.167641 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:15:04.167474 4427 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23bf37a9-9b32-47f9-a4e0-364a7273c02f-insights-runtime-extractor-tls podName:23bf37a9-9b32-47f9-a4e0-364a7273c02f nodeName:}" failed. No retries permitted until 2026-11-02 01:15:12.167468405 +0000 UTC m=+16.788473594 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/23bf37a9-9b32-47f9-a4e0-364a7273c02f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gs62j" (UID: "23bf37a9-9b32-47f9-a4e0-364a7273c02f") : secret "insights-runtime-extractor-tls" not found Nov 02 01:15:04.167678 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:04.167637 4427 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 01:15:04.169890 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:04.169864 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/00d17dbb-cb72-4413-9559-4487c313158c-metrics-tls\") pod \"dns-default-xcm6b\" (UID: \"00d17dbb-cb72-4413-9559-4487c313158c\") " pod="openshift-dns/dns-default-xcm6b" Nov 02 01:15:04.264034 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:04.264002 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:15:04.264208 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:04.264201 4427 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 01:15:04.289318 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:04.289286 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xcm6b" Nov 02 01:15:04.564775 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:15:04.564734 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00d17dbb_cb72_4413_9559_4487c313158c.slice/crio-bef57d1fe64663d4415d556f694556f8a8cca2d007f7e55580f990eebd14804e WatchSource:0}: Error finding container bef57d1fe64663d4415d556f694556f8a8cca2d007f7e55580f990eebd14804e: Status 404 returned error can't find the container with id bef57d1fe64663d4415d556f694556f8a8cca2d007f7e55580f990eebd14804e Nov 02 01:15:04.565778 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:04.565755 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xcm6b"] Nov 02 01:15:04.566831 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:04.566817 4427 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 01:15:04.579893 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:04.579873 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bgczm_41f79a83-9886-4c59-a904-328463dfe674/node-ca/0.log" Nov 02 01:15:04.581927 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:04.581906 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xcm6b" event={"ID":"00d17dbb-cb72-4413-9559-4487c313158c","Type":"ContainerStarted","Data":"bef57d1fe64663d4415d556f694556f8a8cca2d007f7e55580f990eebd14804e"} Nov 02 01:15:05.580533 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:05.580500 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pn4kl_a75a9f45-7678-4014-92d9-93d3f5b260ba/serve-healthcheck-canary/0.log" Nov 02 01:15:05.981023 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:05.980994 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-ctstx_212ace3e-7830-4810-938d-22fd7f251b25/migrator/0.log" Nov 02 01:15:06.484523 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:06.483829 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z8bzg" Nov 02 01:15:07.592421 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:07.592365 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xcm6b" event={"ID":"00d17dbb-cb72-4413-9559-4487c313158c","Type":"ContainerStarted","Data":"3f513715d674b4e5239ca841eda4697363e5aeb7d754ed2ee4cb0e0a911703b3"} Nov 02 01:15:07.592861 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:07.592437 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xcm6b" event={"ID":"00d17dbb-cb72-4413-9559-4487c313158c","Type":"ContainerStarted","Data":"87589a6ccfae8753d2aecf886ae400dd9892cbf96d299b08d70dfff42eca7157"} Nov 02 01:15:07.592861 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:07.592524 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xcm6b" Nov 02 01:15:12.218535 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:12.218500 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-596688686c-rwtbd"] Nov 02 01:15:12.221789 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:12.221767 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-596688686c-rwtbd" Nov 02 01:15:12.227842 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:12.227715 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c7b5fa9-d0b9-4147-bc52-bdd444a2c081-metrics-certs\") pod \"network-metrics-daemon-nkqjm\" (UID: \"4c7b5fa9-d0b9-4147-bc52-bdd444a2c081\") " pod="openshift-multus/network-metrics-daemon-nkqjm" Nov 02 01:15:12.227842 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:12.227754 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/23bf37a9-9b32-47f9-a4e0-364a7273c02f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gs62j\" (UID: \"23bf37a9-9b32-47f9-a4e0-364a7273c02f\") " pod="openshift-insights/insights-runtime-extractor-gs62j" Nov 02 01:15:12.230314 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:12.230282 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/23bf37a9-9b32-47f9-a4e0-364a7273c02f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gs62j\" (UID: \"23bf37a9-9b32-47f9-a4e0-364a7273c02f\") " pod="openshift-insights/insights-runtime-extractor-gs62j" Nov 02 01:15:12.230534 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:12.230517 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 02 01:15:12.230620 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:12.230607 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c7b5fa9-d0b9-4147-bc52-bdd444a2c081-metrics-certs\") pod \"network-metrics-daemon-nkqjm\" (UID: \"4c7b5fa9-d0b9-4147-bc52-bdd444a2c081\") " pod="openshift-multus/network-metrics-daemon-nkqjm" Nov 02 01:15:12.233356 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:12.233304 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 02 01:15:12.233579 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:12.233496 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 02 01:15:12.233579 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:12.233515 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-2zfq9\"" Nov 02 01:15:12.234352 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:12.234320 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-596688686c-rwtbd"] Nov 02 01:15:12.234710 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:12.234693 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 02 01:15:12.328087 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:12.328050 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/872bde0f-dfbc-4226-928f-8c6dfd95281a-ca-trust-extracted\") pod \"image-registry-596688686c-rwtbd\" (UID: \"872bde0f-dfbc-4226-928f-8c6dfd95281a\") " pod="openshift-image-registry/image-registry-596688686c-rwtbd" Nov 02 01:15:12.328087 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:12.328086 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/872bde0f-dfbc-4226-928f-8c6dfd95281a-registry-certificates\") pod \"image-registry-596688686c-rwtbd\" (UID: \"872bde0f-dfbc-4226-928f-8c6dfd95281a\") " pod="openshift-image-registry/image-registry-596688686c-rwtbd" Nov 02 01:15:12.328350 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:12.328106 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spkkw\" (UniqueName: \"kubernetes.io/projected/872bde0f-dfbc-4226-928f-8c6dfd95281a-kube-api-access-spkkw\") pod \"image-registry-596688686c-rwtbd\" (UID: \"872bde0f-dfbc-4226-928f-8c6dfd95281a\") " pod="openshift-image-registry/image-registry-596688686c-rwtbd" Nov 02 01:15:12.328350 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:12.328159 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/872bde0f-dfbc-4226-928f-8c6dfd95281a-installation-pull-secrets\") pod \"image-registry-596688686c-rwtbd\" (UID: \"872bde0f-dfbc-4226-928f-8c6dfd95281a\") " pod="openshift-image-registry/image-registry-596688686c-rwtbd" Nov 02 01:15:12.328350 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:12.328294 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/872bde0f-dfbc-4226-928f-8c6dfd95281a-trusted-ca\") pod \"image-registry-596688686c-rwtbd\" (UID: \"872bde0f-dfbc-4226-928f-8c6dfd95281a\") " pod="openshift-image-registry/image-registry-596688686c-rwtbd" Nov 02 01:15:12.328508 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:12.328349 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/872bde0f-dfbc-4226-928f-8c6dfd95281a-image-registry-private-configuration\") pod \"image-registry-596688686c-rwtbd\" (UID: \"872bde0f-dfbc-4226-928f-8c6dfd95281a\") " pod="openshift-image-registry/image-registry-596688686c-rwtbd" Nov 02 01:15:12.328508 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:12.328418 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/872bde0f-dfbc-4226-928f-8c6dfd95281a-registry-tls\") pod \"image-registry-596688686c-rwtbd\" (UID: \"872bde0f-dfbc-4226-928f-8c6dfd95281a\") " pod="openshift-image-registry/image-registry-596688686c-rwtbd" Nov 02 01:15:12.328508 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:12.328479 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/872bde0f-dfbc-4226-928f-8c6dfd95281a-bound-sa-token\") pod \"image-registry-596688686c-rwtbd\" (UID: \"872bde0f-dfbc-4226-928f-8c6dfd95281a\") " pod="openshift-image-registry/image-registry-596688686c-rwtbd" Nov 02 01:15:12.388235 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:12.388188 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkqjm" Nov 02 01:15:12.390397 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:12.390364 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gs62j" Nov 02 01:15:12.429051 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:12.429014 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/872bde0f-dfbc-4226-928f-8c6dfd95281a-trusted-ca\") pod \"image-registry-596688686c-rwtbd\" (UID: \"872bde0f-dfbc-4226-928f-8c6dfd95281a\") " pod="openshift-image-registry/image-registry-596688686c-rwtbd" Nov 02 01:15:12.429460 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:12.429068 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/872bde0f-dfbc-4226-928f-8c6dfd95281a-image-registry-private-configuration\") pod \"image-registry-596688686c-rwtbd\" (UID: \"872bde0f-dfbc-4226-928f-8c6dfd95281a\") " pod="openshift-image-registry/image-registry-596688686c-rwtbd" Nov 02 01:15:12.429460 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:12.429093 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/872bde0f-dfbc-4226-928f-8c6dfd95281a-registry-tls\") pod \"image-registry-596688686c-rwtbd\" (UID: \"872bde0f-dfbc-4226-928f-8c6dfd95281a\") " pod="openshift-image-registry/image-registry-596688686c-rwtbd" Nov 02 01:15:12.429460 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:12.429116 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/872bde0f-dfbc-4226-928f-8c6dfd95281a-bound-sa-token\") pod \"image-registry-596688686c-rwtbd\" (UID: \"872bde0f-dfbc-4226-928f-8c6dfd95281a\") " pod="openshift-image-registry/image-registry-596688686c-rwtbd" Nov 02 01:15:12.429460 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:12.429147 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/872bde0f-dfbc-4226-928f-8c6dfd95281a-ca-trust-extracted\") pod \"image-registry-596688686c-rwtbd\" (UID: \"872bde0f-dfbc-4226-928f-8c6dfd95281a\") " pod="openshift-image-registry/image-registry-596688686c-rwtbd" Nov 02 01:15:12.429716 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:12.429477 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/872bde0f-dfbc-4226-928f-8c6dfd95281a-registry-certificates\") pod \"image-registry-596688686c-rwtbd\" (UID: \"872bde0f-dfbc-4226-928f-8c6dfd95281a\") " pod="openshift-image-registry/image-registry-596688686c-rwtbd" Nov 02 01:15:12.429716 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:12.429520 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spkkw\" (UniqueName: \"kubernetes.io/projected/872bde0f-dfbc-4226-928f-8c6dfd95281a-kube-api-access-spkkw\") pod \"image-registry-596688686c-rwtbd\" (UID: \"872bde0f-dfbc-4226-928f-8c6dfd95281a\") " pod="openshift-image-registry/image-registry-596688686c-rwtbd" Nov 02 01:15:12.429716 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:12.429548 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/872bde0f-dfbc-4226-928f-8c6dfd95281a-installation-pull-secrets\") pod \"image-registry-596688686c-rwtbd\" (UID: \"872bde0f-dfbc-4226-928f-8c6dfd95281a\") " pod="openshift-image-registry/image-registry-596688686c-rwtbd" Nov 02 01:15:12.432289 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:12.432236 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/872bde0f-dfbc-4226-928f-8c6dfd95281a-registry-certificates\") pod \"image-registry-596688686c-rwtbd\" (UID: \"872bde0f-dfbc-4226-928f-8c6dfd95281a\") " pod="openshift-image-registry/image-registry-596688686c-rwtbd" Nov 02 01:15:12.432436 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:12.432394 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/872bde0f-dfbc-4226-928f-8c6dfd95281a-trusted-ca\") pod \"image-registry-596688686c-rwtbd\" (UID: \"872bde0f-dfbc-4226-928f-8c6dfd95281a\") " pod="openshift-image-registry/image-registry-596688686c-rwtbd" Nov 02 01:15:12.432926 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:12.432877 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/872bde0f-dfbc-4226-928f-8c6dfd95281a-ca-trust-extracted\") pod \"image-registry-596688686c-rwtbd\" (UID: \"872bde0f-dfbc-4226-928f-8c6dfd95281a\") " pod="openshift-image-registry/image-registry-596688686c-rwtbd" Nov 02 01:15:12.433073 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:12.433050 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/872bde0f-dfbc-4226-928f-8c6dfd95281a-installation-pull-secrets\") pod \"image-registry-596688686c-rwtbd\" (UID: \"872bde0f-dfbc-4226-928f-8c6dfd95281a\") " pod="openshift-image-registry/image-registry-596688686c-rwtbd" Nov 02 01:15:12.433355 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:12.433336 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/872bde0f-dfbc-4226-928f-8c6dfd95281a-image-registry-private-configuration\") pod \"image-registry-596688686c-rwtbd\" (UID: \"872bde0f-dfbc-4226-928f-8c6dfd95281a\") " pod="openshift-image-registry/image-registry-596688686c-rwtbd" Nov 02 01:15:12.436416 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:12.436366 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/872bde0f-dfbc-4226-928f-8c6dfd95281a-registry-tls\") pod \"image-registry-596688686c-rwtbd\" (UID: \"872bde0f-dfbc-4226-928f-8c6dfd95281a\") " pod="openshift-image-registry/image-registry-596688686c-rwtbd" Nov 02 01:15:12.446704 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:12.446634 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spkkw\" (UniqueName: \"kubernetes.io/projected/872bde0f-dfbc-4226-928f-8c6dfd95281a-kube-api-access-spkkw\") pod \"image-registry-596688686c-rwtbd\" (UID: \"872bde0f-dfbc-4226-928f-8c6dfd95281a\") " pod="openshift-image-registry/image-registry-596688686c-rwtbd" Nov 02 01:15:12.447578 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:12.447552 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/872bde0f-dfbc-4226-928f-8c6dfd95281a-bound-sa-token\") pod \"image-registry-596688686c-rwtbd\" (UID: \"872bde0f-dfbc-4226-928f-8c6dfd95281a\") " pod="openshift-image-registry/image-registry-596688686c-rwtbd" Nov 02 01:15:12.533797 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:12.533700 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-596688686c-rwtbd" Nov 02 01:15:12.662316 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:12.662272 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nkqjm"] Nov 02 01:15:12.665521 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:12.665492 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gs62j"] Nov 02 01:15:12.665889 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:15:12.665868 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c7b5fa9_d0b9_4147_bc52_bdd444a2c081.slice/crio-5bafad421018b05f3350a844b08bbb3ef641d0a9986d4040c53a49f6b23e096b WatchSource:0}: Error finding container 5bafad421018b05f3350a844b08bbb3ef641d0a9986d4040c53a49f6b23e096b: Status 404 returned error can't find the container with id 5bafad421018b05f3350a844b08bbb3ef641d0a9986d4040c53a49f6b23e096b Nov 02 01:15:12.668093 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:15:12.668065 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23bf37a9_9b32_47f9_a4e0_364a7273c02f.slice/crio-2b624e93927bcf332d9f4ef87e91f10db570f61bc7d0ecad8449360b25d47484 WatchSource:0}: Error finding container 2b624e93927bcf332d9f4ef87e91f10db570f61bc7d0ecad8449360b25d47484: Status 404 returned error can't find the container with id 2b624e93927bcf332d9f4ef87e91f10db570f61bc7d0ecad8449360b25d47484 Nov 02 01:15:12.807523 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:12.807444 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-596688686c-rwtbd"] Nov 02 01:15:12.810291 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:15:12.810263 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod872bde0f_dfbc_4226_928f_8c6dfd95281a.slice/crio-7e0052a1937e6b83d60279795d91fcedc18ba3f058f0f7b0759023e185601184 WatchSource:0}: Error finding container 7e0052a1937e6b83d60279795d91fcedc18ba3f058f0f7b0759023e185601184: Status 404 returned error can't find the container with id 7e0052a1937e6b83d60279795d91fcedc18ba3f058f0f7b0759023e185601184 Nov 02 01:15:13.608403 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:13.607895 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-596688686c-rwtbd" event={"ID":"872bde0f-dfbc-4226-928f-8c6dfd95281a","Type":"ContainerStarted","Data":"44094307c0cecfaee70031f20b07f6974b356258ae15d95d1a14f4b6d8d8b487"} Nov 02 01:15:13.608403 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:13.607958 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-596688686c-rwtbd" event={"ID":"872bde0f-dfbc-4226-928f-8c6dfd95281a","Type":"ContainerStarted","Data":"7e0052a1937e6b83d60279795d91fcedc18ba3f058f0f7b0759023e185601184"} Nov 02 01:15:13.608403 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:13.608023 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-596688686c-rwtbd" Nov 02 01:15:13.609319 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:13.609292 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nkqjm" event={"ID":"4c7b5fa9-d0b9-4147-bc52-bdd444a2c081","Type":"ContainerStarted","Data":"5bafad421018b05f3350a844b08bbb3ef641d0a9986d4040c53a49f6b23e096b"} Nov 02 01:15:13.611038 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:13.611017 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gs62j" event={"ID":"23bf37a9-9b32-47f9-a4e0-364a7273c02f","Type":"ContainerStarted","Data":"f190b9ff747e025d751b31cba458dc492c03d7cdb4fc30053d2419fe2832284c"} Nov 02 01:15:13.611150 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:13.611064 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gs62j" event={"ID":"23bf37a9-9b32-47f9-a4e0-364a7273c02f","Type":"ContainerStarted","Data":"3f939ad657e0f4aadf6ae448a33564aab06fae42a522ae78b398e2a8f9050e57"} Nov 02 01:15:13.611150 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:13.611090 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gs62j" event={"ID":"23bf37a9-9b32-47f9-a4e0-364a7273c02f","Type":"ContainerStarted","Data":"2b624e93927bcf332d9f4ef87e91f10db570f61bc7d0ecad8449360b25d47484"} Nov 02 01:15:13.633065 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:13.632989 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-596688686c-rwtbd" podStartSLOduration=1.632969391 podStartE2EDuration="1.632969391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:15:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:15:13.631214835 +0000 UTC m=+18.252220057" watchObservedRunningTime="2026-11-02 01:15:13.632969391 +0000 UTC m=+18.253974645" Nov 02 01:15:14.617218 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:14.617179 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nkqjm" event={"ID":"4c7b5fa9-d0b9-4147-bc52-bdd444a2c081","Type":"ContainerStarted","Data":"6c4a99bcec9c2f7f3b2a2e0c50ff58cd3de38c3645601e6c2c6ca8a87eccb500"} Nov 02 01:15:14.617636 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:14.617243 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nkqjm" event={"ID":"4c7b5fa9-d0b9-4147-bc52-bdd444a2c081","Type":"ContainerStarted","Data":"bb7cdd839f99ea009cf5011eda4bc51ad5171e02563d5e2e792bc719cec3d47e"} Nov 02 01:15:14.979231 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:14.979145 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-5229n"] Nov 02 01:15:14.982202 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:14.982180 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-5229n" Nov 02 01:15:14.993469 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:14.993443 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 02 01:15:14.993672 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:14.993656 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 02 01:15:14.993959 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:14.993940 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-5229n"] Nov 02 01:15:14.994460 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:14.994427 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 01:15:14.994826 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:14.994794 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 01:15:14.994826 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:14.994809 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-mrqqv\"" Nov 02 01:15:14.994960 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:14.994905 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 01:15:15.051085 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:15.051043 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7965489f-4e02-4318-ad66-64346010ec0c-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-5229n\" (UID: \"7965489f-4e02-4318-ad66-64346010ec0c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-5229n" Nov 02 01:15:15.051293 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:15.051094 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7965489f-4e02-4318-ad66-64346010ec0c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-5229n\" (UID: \"7965489f-4e02-4318-ad66-64346010ec0c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-5229n" Nov 02 01:15:15.051293 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:15.051118 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pmrz\" (UniqueName: \"kubernetes.io/projected/7965489f-4e02-4318-ad66-64346010ec0c-kube-api-access-7pmrz\") pod \"prometheus-operator-9cd89b74c-5229n\" (UID: \"7965489f-4e02-4318-ad66-64346010ec0c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-5229n" Nov 02 01:15:15.051293 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:15.051236 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7965489f-4e02-4318-ad66-64346010ec0c-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-5229n\" (UID: \"7965489f-4e02-4318-ad66-64346010ec0c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-5229n" Nov 02 01:15:15.151924 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:15.151883 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7965489f-4e02-4318-ad66-64346010ec0c-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-5229n\" (UID: \"7965489f-4e02-4318-ad66-64346010ec0c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-5229n" Nov 02 01:15:15.151924 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:15.151924 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7965489f-4e02-4318-ad66-64346010ec0c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-5229n\" (UID: \"7965489f-4e02-4318-ad66-64346010ec0c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-5229n" Nov 02 01:15:15.152173 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:15.151943 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pmrz\" (UniqueName: \"kubernetes.io/projected/7965489f-4e02-4318-ad66-64346010ec0c-kube-api-access-7pmrz\") pod \"prometheus-operator-9cd89b74c-5229n\" (UID: \"7965489f-4e02-4318-ad66-64346010ec0c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-5229n" Nov 02 01:15:15.152173 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:15.151972 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7965489f-4e02-4318-ad66-64346010ec0c-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-5229n\" (UID: \"7965489f-4e02-4318-ad66-64346010ec0c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-5229n" Nov 02 01:15:15.153118 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:15.153087 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7965489f-4e02-4318-ad66-64346010ec0c-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-5229n\" (UID: \"7965489f-4e02-4318-ad66-64346010ec0c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-5229n" Nov 02 01:15:15.157922 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:15.157888 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7965489f-4e02-4318-ad66-64346010ec0c-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-5229n\" (UID: \"7965489f-4e02-4318-ad66-64346010ec0c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-5229n" Nov 02 01:15:15.158302 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:15.158281 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7965489f-4e02-4318-ad66-64346010ec0c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-5229n\" (UID: \"7965489f-4e02-4318-ad66-64346010ec0c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-5229n" Nov 02 01:15:15.165226 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:15.165197 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pmrz\" (UniqueName: \"kubernetes.io/projected/7965489f-4e02-4318-ad66-64346010ec0c-kube-api-access-7pmrz\") pod \"prometheus-operator-9cd89b74c-5229n\" (UID: \"7965489f-4e02-4318-ad66-64346010ec0c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-5229n" Nov 02 01:15:15.295307 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:15.295225 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-5229n" Nov 02 01:15:15.570476 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:15.570399 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-5229n"] Nov 02 01:15:15.573906 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:15:15.573874 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7965489f_4e02_4318_ad66_64346010ec0c.slice/crio-73a4dbbf3fe1d8696da14fe79eed1ba28052c15cbf4902901e61262fdd33536f WatchSource:0}: Error finding container 73a4dbbf3fe1d8696da14fe79eed1ba28052c15cbf4902901e61262fdd33536f: Status 404 returned error can't find the container with id 73a4dbbf3fe1d8696da14fe79eed1ba28052c15cbf4902901e61262fdd33536f Nov 02 01:15:15.621984 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:15.621941 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-5229n" event={"ID":"7965489f-4e02-4318-ad66-64346010ec0c","Type":"ContainerStarted","Data":"73a4dbbf3fe1d8696da14fe79eed1ba28052c15cbf4902901e61262fdd33536f"} Nov 02 01:15:16.627765 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:16.627732 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gs62j" event={"ID":"23bf37a9-9b32-47f9-a4e0-364a7273c02f","Type":"ContainerStarted","Data":"79ff023e2d3c112d9468b9da06e91b5476eff9d29e900ebf378b9d0f2b02adf4"} Nov 02 01:15:17.596762 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:17.596729 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xcm6b" Nov 02 01:15:17.633812 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:17.633770 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-5229n" event={"ID":"7965489f-4e02-4318-ad66-64346010ec0c","Type":"ContainerStarted","Data":"abf1311b6a25b36c619744cddb9b0ddfa57d5d0fcc649133e1b071ac1b8afd14"} Nov 02 01:15:17.634229 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:17.633838 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-5229n" event={"ID":"7965489f-4e02-4318-ad66-64346010ec0c","Type":"ContainerStarted","Data":"67f372bd5cfa22c013152df9c7e7d9bdf586704cdb98897ce7c415998e7172b1"} Nov 02 01:15:17.653185 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:17.653123 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-5229n" podStartSLOduration=2.329759813 podStartE2EDuration="3.653103483s" totalImagesPullingTime="1.32334367s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:15:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:15:17.652214044 +0000 UTC m=+22.273219256" watchObservedRunningTime="2026-11-02 01:15:17.653103483 +0000 UTC m=+22.274108693" Nov 02 01:15:19.353385 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:19.353350 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-jz8q4"] Nov 02 01:15:19.358171 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:19.358155 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jz8q4" Nov 02 01:15:19.387612 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:19.387572 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 02 01:15:19.387612 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:19.387572 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 02 01:15:19.387835 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:19.387783 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-qjskk\"" Nov 02 01:15:19.388498 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:19.388477 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-jz8q4"] Nov 02 01:15:19.400505 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:19.400480 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-pmztz"] Nov 02 01:15:19.403744 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:19.403728 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pmztz" Nov 02 01:15:19.412703 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:19.412679 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 01:15:19.472300 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:19.472268 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 01:15:19.472464 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:19.472434 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 01:15:19.472464 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:19.472437 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pc7pp\"" Nov 02 01:15:19.479862 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:19.479836 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/592c7dda-a5ec-4783-9960-3968165d30e6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-jz8q4\" (UID: \"592c7dda-a5ec-4783-9960-3968165d30e6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jz8q4" Nov 02 01:15:19.479970 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:19.479871 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m46j8\" (UniqueName: \"kubernetes.io/projected/685b3025-3cb1-4149-8d5f-4271ecdacf99-kube-api-access-m46j8\") pod \"node-exporter-pmztz\" (UID: \"685b3025-3cb1-4149-8d5f-4271ecdacf99\") " pod="openshift-monitoring/node-exporter-pmztz" Nov 02 01:15:19.479970 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:19.479894 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/685b3025-3cb1-4149-8d5f-4271ecdacf99-root\") pod \"node-exporter-pmztz\" (UID: \"685b3025-3cb1-4149-8d5f-4271ecdacf99\") " pod="openshift-monitoring/node-exporter-pmztz" Nov 02 01:15:19.479970 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:19.479925 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/685b3025-3cb1-4149-8d5f-4271ecdacf99-sys\") pod \"node-exporter-pmztz\" (UID: \"685b3025-3cb1-4149-8d5f-4271ecdacf99\") " pod="openshift-monitoring/node-exporter-pmztz" Nov 02 01:15:19.480140 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:19.480056 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/592c7dda-a5ec-4783-9960-3968165d30e6-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-jz8q4\" (UID: \"592c7dda-a5ec-4783-9960-3968165d30e6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jz8q4" Nov 02 01:15:19.480140 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:19.480094 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/685b3025-3cb1-4149-8d5f-4271ecdacf99-node-exporter-tls\") pod \"node-exporter-pmztz\" (UID: \"685b3025-3cb1-4149-8d5f-4271ecdacf99\") " pod="openshift-monitoring/node-exporter-pmztz" Nov 02 01:15:19.480140 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:19.480127 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/685b3025-3cb1-4149-8d5f-4271ecdacf99-node-exporter-textfile\") pod \"node-exporter-pmztz\" (UID: \"685b3025-3cb1-4149-8d5f-4271ecdacf99\") " pod="openshift-monitoring/node-exporter-pmztz" Nov 02 01:15:19.480251 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:19.480232 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/685b3025-3cb1-4149-8d5f-4271ecdacf99-node-exporter-wtmp\") pod \"node-exporter-pmztz\" (UID: \"685b3025-3cb1-4149-8d5f-4271ecdacf99\") " pod="openshift-monitoring/node-exporter-pmztz" Nov 02 01:15:19.480294 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:19.480267 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/685b3025-3cb1-4149-8d5f-4271ecdacf99-node-exporter-accelerators-collector-config\") pod \"node-exporter-pmztz\" (UID: \"685b3025-3cb1-4149-8d5f-4271ecdacf99\") " pod="openshift-monitoring/node-exporter-pmztz" Nov 02 01:15:19.480294 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:19.480290 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/685b3025-3cb1-4149-8d5f-4271ecdacf99-metrics-client-ca\") pod \"node-exporter-pmztz\" (UID: \"685b3025-3cb1-4149-8d5f-4271ecdacf99\") " pod="openshift-monitoring/node-exporter-pmztz" Nov 02 01:15:19.480391 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:19.480305 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/685b3025-3cb1-4149-8d5f-4271ecdacf99-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pmztz\" (UID: \"685b3025-3cb1-4149-8d5f-4271ecdacf99\") " pod="openshift-monitoring/node-exporter-pmztz" Nov 02 01:15:19.480391 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:19.480322 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/592c7dda-a5ec-4783-9960-3968165d30e6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-jz8q4\" (UID: \"592c7dda-a5ec-4783-9960-3968165d30e6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jz8q4" Nov 02 01:15:19.480468 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:19.480426 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg5bp\" (UniqueName: \"kubernetes.io/projected/592c7dda-a5ec-4783-9960-3968165d30e6-kube-api-access-dg5bp\") pod \"openshift-state-metrics-7bd7489957-jz8q4\" (UID: \"592c7dda-a5ec-4783-9960-3968165d30e6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jz8q4" Nov 02 01:15:19.581359 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:19.581322 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg5bp\" (UniqueName: \"kubernetes.io/projected/592c7dda-a5ec-4783-9960-3968165d30e6-kube-api-access-dg5bp\") pod \"openshift-state-metrics-7bd7489957-jz8q4\" (UID: \"592c7dda-a5ec-4783-9960-3968165d30e6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jz8q4" Nov 02 01:15:19.581359 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:19.581362 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/592c7dda-a5ec-4783-9960-3968165d30e6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-jz8q4\" (UID: \"592c7dda-a5ec-4783-9960-3968165d30e6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jz8q4" Nov 02 01:15:19.581593 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:19.581408 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m46j8\" (UniqueName: \"kubernetes.io/projected/685b3025-3cb1-4149-8d5f-4271ecdacf99-kube-api-access-m46j8\") pod \"node-exporter-pmztz\" (UID: \"685b3025-3cb1-4149-8d5f-4271ecdacf99\") " pod="openshift-monitoring/node-exporter-pmztz" Nov 02 01:15:19.581593 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:19.581433 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/685b3025-3cb1-4149-8d5f-4271ecdacf99-root\") pod \"node-exporter-pmztz\" (UID: \"685b3025-3cb1-4149-8d5f-4271ecdacf99\") " pod="openshift-monitoring/node-exporter-pmztz" Nov 02 01:15:19.581593 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:19.581464 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/685b3025-3cb1-4149-8d5f-4271ecdacf99-sys\") pod \"node-exporter-pmztz\" (UID: \"685b3025-3cb1-4149-8d5f-4271ecdacf99\") " pod="openshift-monitoring/node-exporter-pmztz" Nov 02 01:15:19.581593 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:19.581518 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/685b3025-3cb1-4149-8d5f-4271ecdacf99-sys\") pod \"node-exporter-pmztz\" (UID: \"685b3025-3cb1-4149-8d5f-4271ecdacf99\") " pod="openshift-monitoring/node-exporter-pmztz" Nov 02 01:15:19.581593 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:19.581541 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/685b3025-3cb1-4149-8d5f-4271ecdacf99-root\") pod \"node-exporter-pmztz\" (UID: \"685b3025-3cb1-4149-8d5f-4271ecdacf99\") " pod="openshift-monitoring/node-exporter-pmztz" Nov 02 01:15:19.581593 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:19.581590 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/592c7dda-a5ec-4783-9960-3968165d30e6-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-jz8q4\" (UID: \"592c7dda-a5ec-4783-9960-3968165d30e6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jz8q4" Nov 02 01:15:19.581863 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:19.581615 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/685b3025-3cb1-4149-8d5f-4271ecdacf99-node-exporter-tls\") pod \"node-exporter-pmztz\" (UID: \"685b3025-3cb1-4149-8d5f-4271ecdacf99\") " pod="openshift-monitoring/node-exporter-pmztz" Nov 02 01:15:19.581863 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:19.581635 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/685b3025-3cb1-4149-8d5f-4271ecdacf99-node-exporter-textfile\") pod \"node-exporter-pmztz\" (UID: \"685b3025-3cb1-4149-8d5f-4271ecdacf99\") " pod="openshift-monitoring/node-exporter-pmztz" Nov 02 01:15:19.581863 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:19.581668 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/685b3025-3cb1-4149-8d5f-4271ecdacf99-node-exporter-wtmp\") pod \"node-exporter-pmztz\" (UID: \"685b3025-3cb1-4149-8d5f-4271ecdacf99\") " pod="openshift-monitoring/node-exporter-pmztz" Nov 02 01:15:19.581863 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:19.581691 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/685b3025-3cb1-4149-8d5f-4271ecdacf99-node-exporter-accelerators-collector-config\") pod \"node-exporter-pmztz\" (UID: \"685b3025-3cb1-4149-8d5f-4271ecdacf99\") " pod="openshift-monitoring/node-exporter-pmztz" Nov 02 01:15:19.581863 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:19.581722 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/685b3025-3cb1-4149-8d5f-4271ecdacf99-metrics-client-ca\") pod \"node-exporter-pmztz\" (UID: \"685b3025-3cb1-4149-8d5f-4271ecdacf99\") " pod="openshift-monitoring/node-exporter-pmztz" Nov 02 01:15:19.581863 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:19.581741 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/685b3025-3cb1-4149-8d5f-4271ecdacf99-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pmztz\" (UID: \"685b3025-3cb1-4149-8d5f-4271ecdacf99\") " pod="openshift-monitoring/node-exporter-pmztz" Nov 02 01:15:19.581863 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:15:19.581756 4427 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Nov 02 01:15:19.581863 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:19.581766 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/592c7dda-a5ec-4783-9960-3968165d30e6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-jz8q4\" (UID: \"592c7dda-a5ec-4783-9960-3968165d30e6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jz8q4" Nov 02 01:15:19.581863 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:15:19.581814 4427 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/685b3025-3cb1-4149-8d5f-4271ecdacf99-node-exporter-tls podName:685b3025-3cb1-4149-8d5f-4271ecdacf99 nodeName:}" failed. No retries permitted until 2026-11-02 01:15:20.081795981 +0000 UTC m=+24.702801170 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/685b3025-3cb1-4149-8d5f-4271ecdacf99-node-exporter-tls") pod "node-exporter-pmztz" (UID: "685b3025-3cb1-4149-8d5f-4271ecdacf99") : secret "node-exporter-tls" not found Nov 02 01:15:19.581863 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:19.581811 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/685b3025-3cb1-4149-8d5f-4271ecdacf99-node-exporter-wtmp\") pod \"node-exporter-pmztz\" (UID: \"685b3025-3cb1-4149-8d5f-4271ecdacf99\") " pod="openshift-monitoring/node-exporter-pmztz" Nov 02 01:15:19.582351 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:19.582023 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/685b3025-3cb1-4149-8d5f-4271ecdacf99-node-exporter-textfile\") pod \"node-exporter-pmztz\" (UID: \"685b3025-3cb1-4149-8d5f-4271ecdacf99\") " pod="openshift-monitoring/node-exporter-pmztz" Nov 02 01:15:19.582436 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:19.582353 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/685b3025-3cb1-4149-8d5f-4271ecdacf99-node-exporter-accelerators-collector-config\") pod \"node-exporter-pmztz\" (UID: \"685b3025-3cb1-4149-8d5f-4271ecdacf99\") " pod="openshift-monitoring/node-exporter-pmztz" Nov 02 01:15:19.582472 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:19.582433 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/685b3025-3cb1-4149-8d5f-4271ecdacf99-metrics-client-ca\") pod \"node-exporter-pmztz\" (UID: \"685b3025-3cb1-4149-8d5f-4271ecdacf99\") " pod="openshift-monitoring/node-exporter-pmztz" Nov 02 01:15:19.582548 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:19.582527 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/592c7dda-a5ec-4783-9960-3968165d30e6-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-jz8q4\" (UID: \"592c7dda-a5ec-4783-9960-3968165d30e6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jz8q4" Nov 02 01:15:19.584241 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:19.584215 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/592c7dda-a5ec-4783-9960-3968165d30e6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-jz8q4\" (UID: \"592c7dda-a5ec-4783-9960-3968165d30e6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jz8q4" Nov 02 01:15:19.584365 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:19.584274 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/592c7dda-a5ec-4783-9960-3968165d30e6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-jz8q4\" (UID: \"592c7dda-a5ec-4783-9960-3968165d30e6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jz8q4" Nov 02 01:15:19.584490 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:19.584469 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/685b3025-3cb1-4149-8d5f-4271ecdacf99-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pmztz\" (UID: \"685b3025-3cb1-4149-8d5f-4271ecdacf99\") " pod="openshift-monitoring/node-exporter-pmztz" Nov 02 01:15:19.597105 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:19.597082 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m46j8\" (UniqueName: \"kubernetes.io/projected/685b3025-3cb1-4149-8d5f-4271ecdacf99-kube-api-access-m46j8\") pod \"node-exporter-pmztz\" (UID: \"685b3025-3cb1-4149-8d5f-4271ecdacf99\") " pod="openshift-monitoring/node-exporter-pmztz" Nov 02 01:15:19.597249 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:19.597219 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg5bp\" (UniqueName: \"kubernetes.io/projected/592c7dda-a5ec-4783-9960-3968165d30e6-kube-api-access-dg5bp\") pod \"openshift-state-metrics-7bd7489957-jz8q4\" (UID: \"592c7dda-a5ec-4783-9960-3968165d30e6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jz8q4" Nov 02 01:15:19.673955 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:19.673921 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jz8q4" Nov 02 01:15:19.783230 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:19.783199 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vcsls" Nov 02 01:15:19.950179 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:19.950107 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-jz8q4"] Nov 02 01:15:19.952935 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:15:19.952908 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod592c7dda_a5ec_4783_9960_3968165d30e6.slice/crio-6b81d4f87f6f8acb9a35575c7df6c480414e00f754af61b0f5051c4caa2b7b0a WatchSource:0}: Error finding container 6b81d4f87f6f8acb9a35575c7df6c480414e00f754af61b0f5051c4caa2b7b0a: Status 404 returned error can't find the container with id 6b81d4f87f6f8acb9a35575c7df6c480414e00f754af61b0f5051c4caa2b7b0a Nov 02 01:15:20.085663 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:20.085627 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/685b3025-3cb1-4149-8d5f-4271ecdacf99-node-exporter-tls\") pod \"node-exporter-pmztz\" (UID: \"685b3025-3cb1-4149-8d5f-4271ecdacf99\") " pod="openshift-monitoring/node-exporter-pmztz" Nov 02 01:15:20.088162 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:20.088140 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/685b3025-3cb1-4149-8d5f-4271ecdacf99-node-exporter-tls\") pod \"node-exporter-pmztz\" (UID: \"685b3025-3cb1-4149-8d5f-4271ecdacf99\") " pod="openshift-monitoring/node-exporter-pmztz" Nov 02 01:15:20.319761 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:20.319670 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pmztz" Nov 02 01:15:20.334982 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:15:20.334941 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod685b3025_3cb1_4149_8d5f_4271ecdacf99.slice/crio-f9d48499f9d420ddf76e15dacbd88d75c7436489d834a1ac22c3e559b73e0cf3 WatchSource:0}: Error finding container f9d48499f9d420ddf76e15dacbd88d75c7436489d834a1ac22c3e559b73e0cf3: Status 404 returned error can't find the container with id f9d48499f9d420ddf76e15dacbd88d75c7436489d834a1ac22c3e559b73e0cf3 Nov 02 01:15:20.646254 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:20.646217 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jz8q4" event={"ID":"592c7dda-a5ec-4783-9960-3968165d30e6","Type":"ContainerStarted","Data":"96f26d6b52e2a5949b0291dc93015dc02aa165e541378af4c6ce31c1d8e63aad"} Nov 02 01:15:20.646665 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:20.646302 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jz8q4" event={"ID":"592c7dda-a5ec-4783-9960-3968165d30e6","Type":"ContainerStarted","Data":"3e17a04fc5c087afc43ac6a1a0dd67f827d7b2f8c7af989b88c290d2e9fd5f69"} Nov 02 01:15:20.646665 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:20.646341 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jz8q4" event={"ID":"592c7dda-a5ec-4783-9960-3968165d30e6","Type":"ContainerStarted","Data":"6b81d4f87f6f8acb9a35575c7df6c480414e00f754af61b0f5051c4caa2b7b0a"} Nov 02 01:15:20.647444 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:20.647421 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pmztz" event={"ID":"685b3025-3cb1-4149-8d5f-4271ecdacf99","Type":"ContainerStarted","Data":"f9d48499f9d420ddf76e15dacbd88d75c7436489d834a1ac22c3e559b73e0cf3"} Nov 02 01:15:21.368229 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:21.368202 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5d6794dd76-m26ph"] Nov 02 01:15:21.375075 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:21.375047 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5d6794dd76-m26ph" Nov 02 01:15:21.383430 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:21.382110 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 02 01:15:21.383430 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:21.382258 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 02 01:15:21.383430 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:21.383393 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-d3c16i5dl0jqg\"" Nov 02 01:15:21.383597 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:21.383459 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 02 01:15:21.383597 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:21.383493 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 02 01:15:21.383597 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:21.383569 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-cqjp8\"" Nov 02 01:15:21.383798 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:21.383606 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 02 01:15:21.386940 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:21.386918 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5d6794dd76-m26ph"] Nov 02 01:15:21.497900 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:21.497865 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/30927eb5-dc3e-46e8-9211-a9fa05d314cd-metrics-client-ca\") pod \"thanos-querier-5d6794dd76-m26ph\" (UID: \"30927eb5-dc3e-46e8-9211-a9fa05d314cd\") " pod="openshift-monitoring/thanos-querier-5d6794dd76-m26ph" Nov 02 01:15:21.498057 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:21.497910 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/30927eb5-dc3e-46e8-9211-a9fa05d314cd-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5d6794dd76-m26ph\" (UID: \"30927eb5-dc3e-46e8-9211-a9fa05d314cd\") " pod="openshift-monitoring/thanos-querier-5d6794dd76-m26ph" Nov 02 01:15:21.498057 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:21.497950 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjdfn\" (UniqueName: \"kubernetes.io/projected/30927eb5-dc3e-46e8-9211-a9fa05d314cd-kube-api-access-gjdfn\") pod \"thanos-querier-5d6794dd76-m26ph\" (UID: \"30927eb5-dc3e-46e8-9211-a9fa05d314cd\") " pod="openshift-monitoring/thanos-querier-5d6794dd76-m26ph" Nov 02 01:15:21.498141 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:21.498075 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/30927eb5-dc3e-46e8-9211-a9fa05d314cd-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5d6794dd76-m26ph\" (UID: \"30927eb5-dc3e-46e8-9211-a9fa05d314cd\") " pod="openshift-monitoring/thanos-querier-5d6794dd76-m26ph" Nov 02 01:15:21.498141 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:21.498114 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/30927eb5-dc3e-46e8-9211-a9fa05d314cd-tmp\") pod \"thanos-querier-5d6794dd76-m26ph\" (UID: \"30927eb5-dc3e-46e8-9211-a9fa05d314cd\") " pod="openshift-monitoring/thanos-querier-5d6794dd76-m26ph" Nov 02 01:15:21.498196 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:21.498147 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/30927eb5-dc3e-46e8-9211-a9fa05d314cd-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5d6794dd76-m26ph\" (UID: \"30927eb5-dc3e-46e8-9211-a9fa05d314cd\") " pod="openshift-monitoring/thanos-querier-5d6794dd76-m26ph" Nov 02 01:15:21.498241 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:21.498220 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/30927eb5-dc3e-46e8-9211-a9fa05d314cd-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5d6794dd76-m26ph\" (UID: \"30927eb5-dc3e-46e8-9211-a9fa05d314cd\") " pod="openshift-monitoring/thanos-querier-5d6794dd76-m26ph" Nov 02 01:15:21.498323 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:21.498271 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/30927eb5-dc3e-46e8-9211-a9fa05d314cd-secret-thanos-querier-tls\") pod \"thanos-querier-5d6794dd76-m26ph\" (UID: \"30927eb5-dc3e-46e8-9211-a9fa05d314cd\") " pod="openshift-monitoring/thanos-querier-5d6794dd76-m26ph" Nov 02 01:15:21.498478 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:21.498338 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/30927eb5-dc3e-46e8-9211-a9fa05d314cd-secret-grpc-tls\") pod \"thanos-querier-5d6794dd76-m26ph\" (UID: \"30927eb5-dc3e-46e8-9211-a9fa05d314cd\") " pod="openshift-monitoring/thanos-querier-5d6794dd76-m26ph" Nov 02 01:15:21.599526 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:21.599344 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/30927eb5-dc3e-46e8-9211-a9fa05d314cd-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5d6794dd76-m26ph\" (UID: \"30927eb5-dc3e-46e8-9211-a9fa05d314cd\") " pod="openshift-monitoring/thanos-querier-5d6794dd76-m26ph" Nov 02 01:15:21.599526 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:21.599421 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/30927eb5-dc3e-46e8-9211-a9fa05d314cd-tmp\") pod \"thanos-querier-5d6794dd76-m26ph\" (UID: \"30927eb5-dc3e-46e8-9211-a9fa05d314cd\") " pod="openshift-monitoring/thanos-querier-5d6794dd76-m26ph" Nov 02 01:15:21.599526 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:21.599467 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/30927eb5-dc3e-46e8-9211-a9fa05d314cd-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5d6794dd76-m26ph\" (UID: \"30927eb5-dc3e-46e8-9211-a9fa05d314cd\") " pod="openshift-monitoring/thanos-querier-5d6794dd76-m26ph" Nov 02 01:15:21.599526 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:21.599498 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/30927eb5-dc3e-46e8-9211-a9fa05d314cd-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5d6794dd76-m26ph\" (UID: \"30927eb5-dc3e-46e8-9211-a9fa05d314cd\") " pod="openshift-monitoring/thanos-querier-5d6794dd76-m26ph" Nov 02 01:15:21.599828 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:21.599534 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/30927eb5-dc3e-46e8-9211-a9fa05d314cd-secret-thanos-querier-tls\") pod \"thanos-querier-5d6794dd76-m26ph\" (UID: \"30927eb5-dc3e-46e8-9211-a9fa05d314cd\") " pod="openshift-monitoring/thanos-querier-5d6794dd76-m26ph" Nov 02 01:15:21.599828 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:21.599560 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/30927eb5-dc3e-46e8-9211-a9fa05d314cd-secret-grpc-tls\") pod \"thanos-querier-5d6794dd76-m26ph\" (UID: \"30927eb5-dc3e-46e8-9211-a9fa05d314cd\") " pod="openshift-monitoring/thanos-querier-5d6794dd76-m26ph" Nov 02 01:15:21.599828 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:21.599590 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/30927eb5-dc3e-46e8-9211-a9fa05d314cd-metrics-client-ca\") pod \"thanos-querier-5d6794dd76-m26ph\" (UID: \"30927eb5-dc3e-46e8-9211-a9fa05d314cd\") " pod="openshift-monitoring/thanos-querier-5d6794dd76-m26ph" Nov 02 01:15:21.599828 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:21.599620 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/30927eb5-dc3e-46e8-9211-a9fa05d314cd-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5d6794dd76-m26ph\" (UID: \"30927eb5-dc3e-46e8-9211-a9fa05d314cd\") " pod="openshift-monitoring/thanos-querier-5d6794dd76-m26ph" Nov 02 01:15:21.599828 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:21.599657 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjdfn\" (UniqueName: \"kubernetes.io/projected/30927eb5-dc3e-46e8-9211-a9fa05d314cd-kube-api-access-gjdfn\") pod \"thanos-querier-5d6794dd76-m26ph\" (UID: \"30927eb5-dc3e-46e8-9211-a9fa05d314cd\") " pod="openshift-monitoring/thanos-querier-5d6794dd76-m26ph" Nov 02 01:15:21.600051 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:21.599858 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/30927eb5-dc3e-46e8-9211-a9fa05d314cd-tmp\") pod \"thanos-querier-5d6794dd76-m26ph\" (UID: \"30927eb5-dc3e-46e8-9211-a9fa05d314cd\") " pod="openshift-monitoring/thanos-querier-5d6794dd76-m26ph" Nov 02 01:15:21.600568 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:21.600539 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/30927eb5-dc3e-46e8-9211-a9fa05d314cd-metrics-client-ca\") pod \"thanos-querier-5d6794dd76-m26ph\" (UID: \"30927eb5-dc3e-46e8-9211-a9fa05d314cd\") " pod="openshift-monitoring/thanos-querier-5d6794dd76-m26ph" Nov 02 01:15:21.602274 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:21.602231 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/30927eb5-dc3e-46e8-9211-a9fa05d314cd-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5d6794dd76-m26ph\" (UID: \"30927eb5-dc3e-46e8-9211-a9fa05d314cd\") " pod="openshift-monitoring/thanos-querier-5d6794dd76-m26ph" Nov 02 01:15:21.602698 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:21.602672 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/30927eb5-dc3e-46e8-9211-a9fa05d314cd-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5d6794dd76-m26ph\" (UID: \"30927eb5-dc3e-46e8-9211-a9fa05d314cd\") " pod="openshift-monitoring/thanos-querier-5d6794dd76-m26ph" Nov 02 01:15:21.602813 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:21.602790 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/30927eb5-dc3e-46e8-9211-a9fa05d314cd-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5d6794dd76-m26ph\" (UID: \"30927eb5-dc3e-46e8-9211-a9fa05d314cd\") " pod="openshift-monitoring/thanos-querier-5d6794dd76-m26ph" Nov 02 01:15:21.602882 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:21.602856 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/30927eb5-dc3e-46e8-9211-a9fa05d314cd-secret-grpc-tls\") pod \"thanos-querier-5d6794dd76-m26ph\" (UID: \"30927eb5-dc3e-46e8-9211-a9fa05d314cd\") " pod="openshift-monitoring/thanos-querier-5d6794dd76-m26ph" Nov 02 01:15:21.603000 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:21.602981 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/30927eb5-dc3e-46e8-9211-a9fa05d314cd-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5d6794dd76-m26ph\" (UID: \"30927eb5-dc3e-46e8-9211-a9fa05d314cd\") " pod="openshift-monitoring/thanos-querier-5d6794dd76-m26ph" Nov 02 01:15:21.603208 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:21.603191 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/30927eb5-dc3e-46e8-9211-a9fa05d314cd-secret-thanos-querier-tls\") pod \"thanos-querier-5d6794dd76-m26ph\" (UID: \"30927eb5-dc3e-46e8-9211-a9fa05d314cd\") " pod="openshift-monitoring/thanos-querier-5d6794dd76-m26ph" Nov 02 01:15:21.611892 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:21.611868 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjdfn\" (UniqueName: \"kubernetes.io/projected/30927eb5-dc3e-46e8-9211-a9fa05d314cd-kube-api-access-gjdfn\") pod \"thanos-querier-5d6794dd76-m26ph\" (UID: \"30927eb5-dc3e-46e8-9211-a9fa05d314cd\") " pod="openshift-monitoring/thanos-querier-5d6794dd76-m26ph" Nov 02 01:15:21.652863 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:21.652836 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jz8q4" event={"ID":"592c7dda-a5ec-4783-9960-3968165d30e6","Type":"ContainerStarted","Data":"9c3d1632edbc8fcb2266870fa4827b9edd6836eefad6ab39a390836c41348d8b"} Nov 02 01:15:21.677623 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:21.677576 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jz8q4" podStartSLOduration=1.421249974 podStartE2EDuration="2.677554428s" totalImagesPullingTime="1.256304454s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:15:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:15:21.676493452 +0000 UTC m=+26.297498664" watchObservedRunningTime="2026-11-02 01:15:21.677554428 +0000 UTC m=+26.298559640" Nov 02 01:15:21.703936 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:21.703899 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5d6794dd76-m26ph" Nov 02 01:15:21.994176 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:21.993990 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5d6794dd76-m26ph"] Nov 02 01:15:21.996700 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:15:21.996667 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30927eb5_dc3e_46e8_9211_a9fa05d314cd.slice/crio-1b6ddcdc39f03e9d35ef575ff5273f470e3b69333c40e7e2263014ac66762c87 WatchSource:0}: Error finding container 1b6ddcdc39f03e9d35ef575ff5273f470e3b69333c40e7e2263014ac66762c87: Status 404 returned error can't find the container with id 1b6ddcdc39f03e9d35ef575ff5273f470e3b69333c40e7e2263014ac66762c87 Nov 02 01:15:22.658769 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:22.658728 4427 generic.go:356] "Generic (PLEG): container finished" podID="685b3025-3cb1-4149-8d5f-4271ecdacf99" containerID="6f1550cfbde7b5fda9c6a16cbbe59e69027383839fc7ae9f4e7d3449c7417fa1" exitCode=0 Nov 02 01:15:22.659247 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:22.658819 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pmztz" event={"ID":"685b3025-3cb1-4149-8d5f-4271ecdacf99","Type":"ContainerDied","Data":"6f1550cfbde7b5fda9c6a16cbbe59e69027383839fc7ae9f4e7d3449c7417fa1"} Nov 02 01:15:22.660313 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:22.660228 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d6794dd76-m26ph" event={"ID":"30927eb5-dc3e-46e8-9211-a9fa05d314cd","Type":"ContainerStarted","Data":"1b6ddcdc39f03e9d35ef575ff5273f470e3b69333c40e7e2263014ac66762c87"} Nov 02 01:15:23.665654 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:23.665621 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pmztz" event={"ID":"685b3025-3cb1-4149-8d5f-4271ecdacf99","Type":"ContainerStarted","Data":"a49a21a0604fc53fe821973ead9252da475e4f0aacd552069e3177b74f367057"} Nov 02 01:15:23.666035 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:23.665668 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pmztz" event={"ID":"685b3025-3cb1-4149-8d5f-4271ecdacf99","Type":"ContainerStarted","Data":"f1262759997e83b48c8fdbeccbeef8ff46da005557666152ad482a1f1a45e433"} Nov 02 01:15:23.689844 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:23.689791 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-pmztz" podStartSLOduration=3.340151639 podStartE2EDuration="4.689775341s" totalImagesPullingTime="1.271275364s" totalInitContainerRuntime="78.348338ms" isStatefulPod=true podCreationTimestamp="2026-11-02 01:15:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:15:23.688410564 +0000 UTC m=+28.309415776" watchObservedRunningTime="2026-11-02 01:15:23.689775341 +0000 UTC m=+28.310780559" Nov 02 01:15:23.847142 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:23.847109 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-fdptg" Nov 02 01:15:25.607237 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.607205 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 01:15:25.646516 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.646482 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:15:25.647430 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.647404 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 01:15:25.657736 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.657711 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 02 01:15:25.657842 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.657749 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 02 01:15:25.657842 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.657837 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 02 01:15:25.658049 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.658033 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 02 01:15:25.658095 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.658074 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 02 01:15:25.658164 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.658041 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 02 01:15:25.658164 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.658149 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 02 01:15:25.658661 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.658642 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 02 01:15:25.658661 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.658657 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 02 01:15:25.659509 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.658743 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-49d8x\"" Nov 02 01:15:25.659509 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.658757 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 02 01:15:25.659509 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.658831 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 02 01:15:25.661849 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.661657 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 02 01:15:25.661985 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.661968 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 02 01:15:25.662225 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.662195 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-26laidsh9cjjh\"" Nov 02 01:15:25.664092 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.663401 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 02 01:15:25.664092 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.663587 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 02 01:15:25.686097 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.686061 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d6794dd76-m26ph" event={"ID":"30927eb5-dc3e-46e8-9211-a9fa05d314cd","Type":"ContainerStarted","Data":"d923e69426b25bc30433f4c0a71d623cb4b2da2c02c73de3d40ca08f0f03dbea"} Nov 02 01:15:25.733324 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.733300 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0472228c-7bb0-46c5-928d-2b0804457bca-web-config\") pod \"prometheus-k8s-0\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:15:25.733438 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.733335 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0472228c-7bb0-46c5-928d-2b0804457bca-config-out\") pod \"prometheus-k8s-0\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:15:25.733438 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.733358 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0472228c-7bb0-46c5-928d-2b0804457bca-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:15:25.733438 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.733398 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0472228c-7bb0-46c5-928d-2b0804457bca-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:15:25.733438 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.733420 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0472228c-7bb0-46c5-928d-2b0804457bca-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:15:25.733438 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.733436 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0472228c-7bb0-46c5-928d-2b0804457bca-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:15:25.733587 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.733452 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0472228c-7bb0-46c5-928d-2b0804457bca-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:15:25.733587 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.733468 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0472228c-7bb0-46c5-928d-2b0804457bca-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:15:25.733587 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.733482 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0472228c-7bb0-46c5-928d-2b0804457bca-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:15:25.733587 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.733500 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0472228c-7bb0-46c5-928d-2b0804457bca-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:15:25.733587 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.733516 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0472228c-7bb0-46c5-928d-2b0804457bca-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:15:25.733587 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.733538 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0472228c-7bb0-46c5-928d-2b0804457bca-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:15:25.733587 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.733558 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0472228c-7bb0-46c5-928d-2b0804457bca-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:15:25.733587 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.733576 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0472228c-7bb0-46c5-928d-2b0804457bca-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:15:25.733842 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.733595 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0472228c-7bb0-46c5-928d-2b0804457bca-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:15:25.733842 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.733615 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0472228c-7bb0-46c5-928d-2b0804457bca-config\") pod \"prometheus-k8s-0\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:15:25.733842 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.733629 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0472228c-7bb0-46c5-928d-2b0804457bca-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:15:25.733842 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.733652 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d8mz\" (UniqueName: \"kubernetes.io/projected/0472228c-7bb0-46c5-928d-2b0804457bca-kube-api-access-8d8mz\") pod \"prometheus-k8s-0\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:15:25.733842 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.733669 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0472228c-7bb0-46c5-928d-2b0804457bca-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:15:25.733842 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.733692 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0472228c-7bb0-46c5-928d-2b0804457bca-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:15:25.834659 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.834631 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0472228c-7bb0-46c5-928d-2b0804457bca-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:15:25.834778 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.834673 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0472228c-7bb0-46c5-928d-2b0804457bca-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:15:25.834778 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.834700 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0472228c-7bb0-46c5-928d-2b0804457bca-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:15:25.834778 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.834745 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0472228c-7bb0-46c5-928d-2b0804457bca-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:15:25.834778 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.834774 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0472228c-7bb0-46c5-928d-2b0804457bca-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:15:25.835047 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.834806 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0472228c-7bb0-46c5-928d-2b0804457bca-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:15:25.835047 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.834836 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0472228c-7bb0-46c5-928d-2b0804457bca-config\") pod \"prometheus-k8s-0\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:15:25.835047 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.834861 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0472228c-7bb0-46c5-928d-2b0804457bca-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:15:25.835047 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.834895 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d8mz\" (UniqueName: \"kubernetes.io/projected/0472228c-7bb0-46c5-928d-2b0804457bca-kube-api-access-8d8mz\") pod \"prometheus-k8s-0\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:15:25.835047 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.834921 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0472228c-7bb0-46c5-928d-2b0804457bca-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:15:25.835047 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.834951 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0472228c-7bb0-46c5-928d-2b0804457bca-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:15:25.835047 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.834982 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0472228c-7bb0-46c5-928d-2b0804457bca-web-config\") pod \"prometheus-k8s-0\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:15:25.835047 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.835021 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0472228c-7bb0-46c5-928d-2b0804457bca-config-out\") pod \"prometheus-k8s-0\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:15:25.835504 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.835064 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0472228c-7bb0-46c5-928d-2b0804457bca-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:15:25.835504 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.835102 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0472228c-7bb0-46c5-928d-2b0804457bca-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:15:25.835504 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.835126 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0472228c-7bb0-46c5-928d-2b0804457bca-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:15:25.835504 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.835161 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0472228c-7bb0-46c5-928d-2b0804457bca-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:15:25.835504 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.835223 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0472228c-7bb0-46c5-928d-2b0804457bca-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:15:25.835504 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.835248 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0472228c-7bb0-46c5-928d-2b0804457bca-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:15:25.835504 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.835274 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0472228c-7bb0-46c5-928d-2b0804457bca-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:15:25.835836 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.835594 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0472228c-7bb0-46c5-928d-2b0804457bca-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:15:25.835836 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.835748 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0472228c-7bb0-46c5-928d-2b0804457bca-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:15:25.836743 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.836717 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0472228c-7bb0-46c5-928d-2b0804457bca-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:15:25.837515 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.836851 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0472228c-7bb0-46c5-928d-2b0804457bca-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:15:25.837515 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.837361 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0472228c-7bb0-46c5-928d-2b0804457bca-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:15:25.840323 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.840300 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0472228c-7bb0-46c5-928d-2b0804457bca-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:15:25.840606 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.840587 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0472228c-7bb0-46c5-928d-2b0804457bca-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:15:25.840966 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.840940 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0472228c-7bb0-46c5-928d-2b0804457bca-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:15:25.843141 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.843116 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0472228c-7bb0-46c5-928d-2b0804457bca-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:15:25.844477 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.843765 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0472228c-7bb0-46c5-928d-2b0804457bca-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:15:25.844477 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.844344 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0472228c-7bb0-46c5-928d-2b0804457bca-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:15:25.844908 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.844887 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0472228c-7bb0-46c5-928d-2b0804457bca-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:15:25.844961 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.844907 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0472228c-7bb0-46c5-928d-2b0804457bca-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:15:25.845338 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.845315 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0472228c-7bb0-46c5-928d-2b0804457bca-config\") pod \"prometheus-k8s-0\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:15:25.845723 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.845701 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0472228c-7bb0-46c5-928d-2b0804457bca-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:15:25.845902 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.845875 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0472228c-7bb0-46c5-928d-2b0804457bca-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:15:25.849794 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.849046 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0472228c-7bb0-46c5-928d-2b0804457bca-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:15:25.856672 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.856634 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0472228c-7bb0-46c5-928d-2b0804457bca-web-config\") pod \"prometheus-k8s-0\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:15:25.856908 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.856891 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d8mz\" (UniqueName: \"kubernetes.io/projected/0472228c-7bb0-46c5-928d-2b0804457bca-kube-api-access-8d8mz\") pod \"prometheus-k8s-0\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:15:25.857123 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:25.857107 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0472228c-7bb0-46c5-928d-2b0804457bca-config-out\") pod \"prometheus-k8s-0\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:15:26.079941 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:26.079857 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:15:26.382643 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:26.382595 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 01:15:26.386263 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:15:26.386237 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0472228c_7bb0_46c5_928d_2b0804457bca.slice/crio-cef0c400f58c0396504168983fa5c8c3e19bf9d720acbfb6454d7bb1dcd520a2 WatchSource:0}: Error finding container cef0c400f58c0396504168983fa5c8c3e19bf9d720acbfb6454d7bb1dcd520a2: Status 404 returned error can't find the container with id cef0c400f58c0396504168983fa5c8c3e19bf9d720acbfb6454d7bb1dcd520a2 Nov 02 01:15:26.691674 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:26.691599 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0472228c-7bb0-46c5-928d-2b0804457bca","Type":"ContainerStarted","Data":"cef0c400f58c0396504168983fa5c8c3e19bf9d720acbfb6454d7bb1dcd520a2"} Nov 02 01:15:26.693819 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:26.693797 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d6794dd76-m26ph" event={"ID":"30927eb5-dc3e-46e8-9211-a9fa05d314cd","Type":"ContainerStarted","Data":"e75cd276625684d7663e7abd9f77fba9df50ba5e0b31ed58da7df4fd84a18cde"} Nov 02 01:15:26.693934 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:26.693857 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d6794dd76-m26ph" event={"ID":"30927eb5-dc3e-46e8-9211-a9fa05d314cd","Type":"ContainerStarted","Data":"c84d4f0dd5be82e4b58db6b901e0ce0da7230bb9eb41ec1ba36d4f6429a71e7c"} Nov 02 01:15:27.701912 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:27.701868 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d6794dd76-m26ph" event={"ID":"30927eb5-dc3e-46e8-9211-a9fa05d314cd","Type":"ContainerStarted","Data":"0d33586b312389f3d7e2badc7d1c4e51f52298f95a0d59612e9b630b7e2e35fc"} Nov 02 01:15:27.702398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:27.701948 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d6794dd76-m26ph" event={"ID":"30927eb5-dc3e-46e8-9211-a9fa05d314cd","Type":"ContainerStarted","Data":"fef19d61eb058cf8f43e40b0b791e9c35d5990ab32356a6860070df97e26e117"} Nov 02 01:15:27.702398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:27.701989 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d6794dd76-m26ph" event={"ID":"30927eb5-dc3e-46e8-9211-a9fa05d314cd","Type":"ContainerStarted","Data":"4776f1bc8a716b4d83f01c4c5a23f04f46aa338a3436f2666d08e7a9bee24e5b"} Nov 02 01:15:27.702398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:27.702152 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5d6794dd76-m26ph" Nov 02 01:15:27.734785 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:27.734686 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5d6794dd76-m26ph" podStartSLOduration=1.730086001 podStartE2EDuration="6.734668975s" totalImagesPullingTime="5.004582974s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:15:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:15:27.732361968 +0000 UTC m=+32.353367179" watchObservedRunningTime="2026-11-02 01:15:27.734668975 +0000 UTC m=+32.355674188" Nov 02 01:15:28.712103 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:28.712015 4427 generic.go:356] "Generic (PLEG): container finished" podID="0472228c-7bb0-46c5-928d-2b0804457bca" containerID="f67e3fcbc2e5b69aef6d52b0047ea0987adfb86dff9b149f9285095c3ffcab70" exitCode=0 Nov 02 01:15:28.712617 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:28.712109 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0472228c-7bb0-46c5-928d-2b0804457bca","Type":"ContainerDied","Data":"f67e3fcbc2e5b69aef6d52b0047ea0987adfb86dff9b149f9285095c3ffcab70"} Nov 02 01:15:31.585351 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:15:31.585314 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:15:33.718496 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:33.718420 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5d6794dd76-m26ph" Nov 02 01:15:33.735273 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:33.735243 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0472228c-7bb0-46c5-928d-2b0804457bca","Type":"ContainerStarted","Data":"79687feb72bf527960e10d6b14f53e17a970b201597bfb56d7f5288f1f3bbca6"} Nov 02 01:15:33.735432 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:33.735307 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0472228c-7bb0-46c5-928d-2b0804457bca","Type":"ContainerStarted","Data":"0f33a858f189cfcca7f3b166f59bd76ce8e71300eb2a80245e2f38b9548f3a3f"} Nov 02 01:15:33.735432 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:33.735335 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0472228c-7bb0-46c5-928d-2b0804457bca","Type":"ContainerStarted","Data":"bc37e69058349568185f4b1203e846b8c7d725831eaf51d34b2f7d2db466d069"} Nov 02 01:15:33.735432 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:33.735359 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0472228c-7bb0-46c5-928d-2b0804457bca","Type":"ContainerStarted","Data":"faaf4b41883cfbc689b20cec311640587bcae25c66591b3b1571086ba983b49c"} Nov 02 01:15:33.735432 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:33.735414 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0472228c-7bb0-46c5-928d-2b0804457bca","Type":"ContainerStarted","Data":"4eab1461e0e4faddd78b5114775fe587b85555eec02233eb20ca1f7e771aad55"} Nov 02 01:15:33.735583 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:33.735444 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0472228c-7bb0-46c5-928d-2b0804457bca","Type":"ContainerStarted","Data":"5a8ea4231e84fc19c98ab495d418e2bdcedb1726c88742c560a480f6b087a515"} Nov 02 01:15:33.783628 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:33.783573 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.3595392 podStartE2EDuration="8.783554013s" totalImagesPullingTime="6.416030206s" totalInitContainerRuntime="7.984607ms" isStatefulPod=false podCreationTimestamp="2026-11-02 01:15:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:15:33.782003832 +0000 UTC m=+38.403009056" watchObservedRunningTime="2026-11-02 01:15:33.783554013 +0000 UTC m=+38.404559231" Nov 02 01:15:34.622850 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:34.622817 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-596688686c-rwtbd" Nov 02 01:15:36.080902 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:36.080865 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:15:44.836588 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:44.836553 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:15:44.855340 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:44.855313 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:15:45.795140 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:15:45.795114 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:01.580282 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:16:01.580239 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:16:13.455620 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.455586 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 01:16:13.456785 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.456728 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0472228c-7bb0-46c5-928d-2b0804457bca" containerName="prometheus" containerID="cri-o://5a8ea4231e84fc19c98ab495d418e2bdcedb1726c88742c560a480f6b087a515" gracePeriod=600 Nov 02 01:16:13.457316 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.457288 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0472228c-7bb0-46c5-928d-2b0804457bca" containerName="kube-rbac-proxy-thanos" containerID="cri-o://79687feb72bf527960e10d6b14f53e17a970b201597bfb56d7f5288f1f3bbca6" gracePeriod=600 Nov 02 01:16:13.457602 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.457545 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0472228c-7bb0-46c5-928d-2b0804457bca" containerName="thanos-sidecar" containerID="cri-o://faaf4b41883cfbc689b20cec311640587bcae25c66591b3b1571086ba983b49c" gracePeriod=600 Nov 02 01:16:13.458264 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.457555 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0472228c-7bb0-46c5-928d-2b0804457bca" containerName="config-reloader" containerID="cri-o://4eab1461e0e4faddd78b5114775fe587b85555eec02233eb20ca1f7e771aad55" gracePeriod=600 Nov 02 01:16:13.458264 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.457587 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0472228c-7bb0-46c5-928d-2b0804457bca" containerName="kube-rbac-proxy-web" containerID="cri-o://bc37e69058349568185f4b1203e846b8c7d725831eaf51d34b2f7d2db466d069" gracePeriod=600 Nov 02 01:16:13.458264 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.457588 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0472228c-7bb0-46c5-928d-2b0804457bca" containerName="kube-rbac-proxy" containerID="cri-o://0f33a858f189cfcca7f3b166f59bd76ce8e71300eb2a80245e2f38b9548f3a3f" gracePeriod=600 Nov 02 01:16:13.712278 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.712212 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:13.825296 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.825252 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0472228c-7bb0-46c5-928d-2b0804457bca-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0472228c-7bb0-46c5-928d-2b0804457bca-secret-prometheus-k8s-tls\") pod \"0472228c-7bb0-46c5-928d-2b0804457bca\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " Nov 02 01:16:13.825296 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.825298 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0472228c-7bb0-46c5-928d-2b0804457bca-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0472228c-7bb0-46c5-928d-2b0804457bca-prometheus-k8s-rulefiles-0\") pod \"0472228c-7bb0-46c5-928d-2b0804457bca\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " Nov 02 01:16:13.825569 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.825329 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0472228c-7bb0-46c5-928d-2b0804457bca-config-out\" (UniqueName: \"kubernetes.io/empty-dir/0472228c-7bb0-46c5-928d-2b0804457bca-config-out\") pod \"0472228c-7bb0-46c5-928d-2b0804457bca\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " Nov 02 01:16:13.825569 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.825511 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0472228c-7bb0-46c5-928d-2b0804457bca-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0472228c-7bb0-46c5-928d-2b0804457bca-prometheus-k8s-db\") pod \"0472228c-7bb0-46c5-928d-2b0804457bca\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " Nov 02 01:16:13.825569 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.825560 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0472228c-7bb0-46c5-928d-2b0804457bca-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0472228c-7bb0-46c5-928d-2b0804457bca-configmap-metrics-client-ca\") pod \"0472228c-7bb0-46c5-928d-2b0804457bca\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " Nov 02 01:16:13.825732 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.825596 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0472228c-7bb0-46c5-928d-2b0804457bca-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0472228c-7bb0-46c5-928d-2b0804457bca-configmap-serving-certs-ca-bundle\") pod \"0472228c-7bb0-46c5-928d-2b0804457bca\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " Nov 02 01:16:13.825732 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.825641 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0472228c-7bb0-46c5-928d-2b0804457bca-kube-api-access-8d8mz\" (UniqueName: \"kubernetes.io/projected/0472228c-7bb0-46c5-928d-2b0804457bca-kube-api-access-8d8mz\") pod \"0472228c-7bb0-46c5-928d-2b0804457bca\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " Nov 02 01:16:13.825732 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.825674 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0472228c-7bb0-46c5-928d-2b0804457bca-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0472228c-7bb0-46c5-928d-2b0804457bca-secret-metrics-client-certs\") pod \"0472228c-7bb0-46c5-928d-2b0804457bca\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " Nov 02 01:16:13.825732 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.825703 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0472228c-7bb0-46c5-928d-2b0804457bca-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0472228c-7bb0-46c5-928d-2b0804457bca-configmap-kubelet-serving-ca-bundle\") pod \"0472228c-7bb0-46c5-928d-2b0804457bca\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " Nov 02 01:16:13.825935 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.825732 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0472228c-7bb0-46c5-928d-2b0804457bca-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0472228c-7bb0-46c5-928d-2b0804457bca-prometheus-trusted-ca-bundle\") pod \"0472228c-7bb0-46c5-928d-2b0804457bca\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " Nov 02 01:16:13.825935 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.825779 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0472228c-7bb0-46c5-928d-2b0804457bca-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0472228c-7bb0-46c5-928d-2b0804457bca-secret-grpc-tls\") pod \"0472228c-7bb0-46c5-928d-2b0804457bca\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " Nov 02 01:16:13.825935 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.825809 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0472228c-7bb0-46c5-928d-2b0804457bca-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0472228c-7bb0-46c5-928d-2b0804457bca-secret-kube-rbac-proxy\") pod \"0472228c-7bb0-46c5-928d-2b0804457bca\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " Nov 02 01:16:13.825935 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.825848 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0472228c-7bb0-46c5-928d-2b0804457bca-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0472228c-7bb0-46c5-928d-2b0804457bca-thanos-prometheus-http-client-file\") pod \"0472228c-7bb0-46c5-928d-2b0804457bca\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " Nov 02 01:16:13.825935 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.825875 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0472228c-7bb0-46c5-928d-2b0804457bca-config\" (UniqueName: \"kubernetes.io/secret/0472228c-7bb0-46c5-928d-2b0804457bca-config\") pod \"0472228c-7bb0-46c5-928d-2b0804457bca\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " Nov 02 01:16:13.825935 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.825927 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0472228c-7bb0-46c5-928d-2b0804457bca-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0472228c-7bb0-46c5-928d-2b0804457bca-prometheus-k8s-rulefiles-2\") pod \"0472228c-7bb0-46c5-928d-2b0804457bca\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " Nov 02 01:16:13.826428 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.825954 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0472228c-7bb0-46c5-928d-2b0804457bca-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0472228c-7bb0-46c5-928d-2b0804457bca-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"0472228c-7bb0-46c5-928d-2b0804457bca\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " Nov 02 01:16:13.826428 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.825963 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0472228c-7bb0-46c5-928d-2b0804457bca-configmap-serving-certs-ca-bundle" pod "0472228c-7bb0-46c5-928d-2b0804457bca" (UID: "0472228c-7bb0-46c5-928d-2b0804457bca"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:16:13.826428 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.825988 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0472228c-7bb0-46c5-928d-2b0804457bca-web-config\" (UniqueName: \"kubernetes.io/secret/0472228c-7bb0-46c5-928d-2b0804457bca-web-config\") pod \"0472228c-7bb0-46c5-928d-2b0804457bca\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " Nov 02 01:16:13.826428 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.826092 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0472228c-7bb0-46c5-928d-2b0804457bca-tls-assets\" (UniqueName: \"kubernetes.io/projected/0472228c-7bb0-46c5-928d-2b0804457bca-tls-assets\") pod \"0472228c-7bb0-46c5-928d-2b0804457bca\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " Nov 02 01:16:13.826428 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.826173 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0472228c-7bb0-46c5-928d-2b0804457bca-configmap-kubelet-serving-ca-bundle" pod "0472228c-7bb0-46c5-928d-2b0804457bca" (UID: "0472228c-7bb0-46c5-928d-2b0804457bca"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:16:13.826428 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.826221 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0472228c-7bb0-46c5-928d-2b0804457bca-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0472228c-7bb0-46c5-928d-2b0804457bca-prometheus-k8s-rulefiles-1\") pod \"0472228c-7bb0-46c5-928d-2b0804457bca\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " Nov 02 01:16:13.826428 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.826255 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0472228c-7bb0-46c5-928d-2b0804457bca-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0472228c-7bb0-46c5-928d-2b0804457bca-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"0472228c-7bb0-46c5-928d-2b0804457bca\" (UID: \"0472228c-7bb0-46c5-928d-2b0804457bca\") " Nov 02 01:16:13.826428 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.826320 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0472228c-7bb0-46c5-928d-2b0804457bca-prometheus-k8s-db" pod "0472228c-7bb0-46c5-928d-2b0804457bca" (UID: "0472228c-7bb0-46c5-928d-2b0804457bca"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:16:13.826836 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.826514 4427 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0472228c-7bb0-46c5-928d-2b0804457bca-prometheus-k8s-db\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:16:13.826836 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.826526 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0472228c-7bb0-46c5-928d-2b0804457bca-configmap-metrics-client-ca" pod "0472228c-7bb0-46c5-928d-2b0804457bca" (UID: "0472228c-7bb0-46c5-928d-2b0804457bca"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:16:13.826836 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.826535 4427 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0472228c-7bb0-46c5-928d-2b0804457bca-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:16:13.826836 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.826569 4427 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0472228c-7bb0-46c5-928d-2b0804457bca-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:16:13.827314 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.827033 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0472228c-7bb0-46c5-928d-2b0804457bca-prometheus-k8s-rulefiles-0" pod "0472228c-7bb0-46c5-928d-2b0804457bca" (UID: "0472228c-7bb0-46c5-928d-2b0804457bca"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:16:13.828742 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.828316 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0472228c-7bb0-46c5-928d-2b0804457bca-prometheus-trusted-ca-bundle" pod "0472228c-7bb0-46c5-928d-2b0804457bca" (UID: "0472228c-7bb0-46c5-928d-2b0804457bca"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:16:13.828742 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.828348 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0472228c-7bb0-46c5-928d-2b0804457bca-prometheus-k8s-rulefiles-2" pod "0472228c-7bb0-46c5-928d-2b0804457bca" (UID: "0472228c-7bb0-46c5-928d-2b0804457bca"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:16:13.828742 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.828699 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0472228c-7bb0-46c5-928d-2b0804457bca-prometheus-k8s-rulefiles-1" pod "0472228c-7bb0-46c5-928d-2b0804457bca" (UID: "0472228c-7bb0-46c5-928d-2b0804457bca"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:16:13.829228 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.829193 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0472228c-7bb0-46c5-928d-2b0804457bca-config-out" pod "0472228c-7bb0-46c5-928d-2b0804457bca" (UID: "0472228c-7bb0-46c5-928d-2b0804457bca"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:16:13.829359 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.829324 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0472228c-7bb0-46c5-928d-2b0804457bca-secret-prometheus-k8s-tls" pod "0472228c-7bb0-46c5-928d-2b0804457bca" (UID: "0472228c-7bb0-46c5-928d-2b0804457bca"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:16:13.829540 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.829507 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0472228c-7bb0-46c5-928d-2b0804457bca-secret-prometheus-k8s-kube-rbac-proxy-web" pod "0472228c-7bb0-46c5-928d-2b0804457bca" (UID: "0472228c-7bb0-46c5-928d-2b0804457bca"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:16:13.829602 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.829584 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0472228c-7bb0-46c5-928d-2b0804457bca-thanos-prometheus-http-client-file" pod "0472228c-7bb0-46c5-928d-2b0804457bca" (UID: "0472228c-7bb0-46c5-928d-2b0804457bca"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:16:13.829646 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.829609 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0472228c-7bb0-46c5-928d-2b0804457bca-secret-prometheus-k8s-thanos-sidecar-tls" pod "0472228c-7bb0-46c5-928d-2b0804457bca" (UID: "0472228c-7bb0-46c5-928d-2b0804457bca"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:16:13.829964 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.829812 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0472228c-7bb0-46c5-928d-2b0804457bca-config" pod "0472228c-7bb0-46c5-928d-2b0804457bca" (UID: "0472228c-7bb0-46c5-928d-2b0804457bca"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:16:13.831328 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.831305 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0472228c-7bb0-46c5-928d-2b0804457bca-kube-api-access-8d8mz" pod "0472228c-7bb0-46c5-928d-2b0804457bca" (UID: "0472228c-7bb0-46c5-928d-2b0804457bca"). InnerVolumeSpecName "kube-api-access-8d8mz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:16:13.831565 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.831532 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0472228c-7bb0-46c5-928d-2b0804457bca-secret-grpc-tls" pod "0472228c-7bb0-46c5-928d-2b0804457bca" (UID: "0472228c-7bb0-46c5-928d-2b0804457bca"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:16:13.831669 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.831562 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0472228c-7bb0-46c5-928d-2b0804457bca-secret-metrics-client-certs" pod "0472228c-7bb0-46c5-928d-2b0804457bca" (UID: "0472228c-7bb0-46c5-928d-2b0804457bca"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:16:13.831669 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.831629 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0472228c-7bb0-46c5-928d-2b0804457bca-tls-assets" pod "0472228c-7bb0-46c5-928d-2b0804457bca" (UID: "0472228c-7bb0-46c5-928d-2b0804457bca"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:16:13.831815 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.831796 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0472228c-7bb0-46c5-928d-2b0804457bca-secret-kube-rbac-proxy" pod "0472228c-7bb0-46c5-928d-2b0804457bca" (UID: "0472228c-7bb0-46c5-928d-2b0804457bca"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:16:13.841250 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.841228 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0472228c-7bb0-46c5-928d-2b0804457bca-web-config" pod "0472228c-7bb0-46c5-928d-2b0804457bca" (UID: "0472228c-7bb0-46c5-928d-2b0804457bca"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:16:13.884519 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.884482 4427 generic.go:356] "Generic (PLEG): container finished" podID="0472228c-7bb0-46c5-928d-2b0804457bca" containerID="79687feb72bf527960e10d6b14f53e17a970b201597bfb56d7f5288f1f3bbca6" exitCode=0 Nov 02 01:16:13.884519 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.884510 4427 generic.go:356] "Generic (PLEG): container finished" podID="0472228c-7bb0-46c5-928d-2b0804457bca" containerID="0f33a858f189cfcca7f3b166f59bd76ce8e71300eb2a80245e2f38b9548f3a3f" exitCode=0 Nov 02 01:16:13.884519 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.884519 4427 generic.go:356] "Generic (PLEG): container finished" podID="0472228c-7bb0-46c5-928d-2b0804457bca" containerID="bc37e69058349568185f4b1203e846b8c7d725831eaf51d34b2f7d2db466d069" exitCode=0 Nov 02 01:16:13.884519 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.884524 4427 generic.go:356] "Generic (PLEG): container finished" podID="0472228c-7bb0-46c5-928d-2b0804457bca" containerID="faaf4b41883cfbc689b20cec311640587bcae25c66591b3b1571086ba983b49c" exitCode=0 Nov 02 01:16:13.884519 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.884529 4427 generic.go:356] "Generic (PLEG): container finished" podID="0472228c-7bb0-46c5-928d-2b0804457bca" containerID="4eab1461e0e4faddd78b5114775fe587b85555eec02233eb20ca1f7e771aad55" exitCode=0 Nov 02 01:16:13.884900 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.884536 4427 generic.go:356] "Generic (PLEG): container finished" podID="0472228c-7bb0-46c5-928d-2b0804457bca" containerID="5a8ea4231e84fc19c98ab495d418e2bdcedb1726c88742c560a480f6b087a515" exitCode=0 Nov 02 01:16:13.884900 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.884552 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0472228c-7bb0-46c5-928d-2b0804457bca","Type":"ContainerDied","Data":"79687feb72bf527960e10d6b14f53e17a970b201597bfb56d7f5288f1f3bbca6"} Nov 02 01:16:13.884900 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.884569 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:13.884900 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.884637 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0472228c-7bb0-46c5-928d-2b0804457bca","Type":"ContainerDied","Data":"0f33a858f189cfcca7f3b166f59bd76ce8e71300eb2a80245e2f38b9548f3a3f"} Nov 02 01:16:13.884900 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.884680 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0472228c-7bb0-46c5-928d-2b0804457bca","Type":"ContainerDied","Data":"bc37e69058349568185f4b1203e846b8c7d725831eaf51d34b2f7d2db466d069"} Nov 02 01:16:13.884900 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.884701 4427 scope.go:122] "RemoveContainer" containerID="79687feb72bf527960e10d6b14f53e17a970b201597bfb56d7f5288f1f3bbca6" Nov 02 01:16:13.884900 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.884720 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0472228c-7bb0-46c5-928d-2b0804457bca","Type":"ContainerDied","Data":"faaf4b41883cfbc689b20cec311640587bcae25c66591b3b1571086ba983b49c"} Nov 02 01:16:13.884900 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.884761 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0472228c-7bb0-46c5-928d-2b0804457bca","Type":"ContainerDied","Data":"4eab1461e0e4faddd78b5114775fe587b85555eec02233eb20ca1f7e771aad55"} Nov 02 01:16:13.884900 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.884801 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0472228c-7bb0-46c5-928d-2b0804457bca","Type":"ContainerDied","Data":"5a8ea4231e84fc19c98ab495d418e2bdcedb1726c88742c560a480f6b087a515"} Nov 02 01:16:13.884900 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.884848 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0472228c-7bb0-46c5-928d-2b0804457bca","Type":"ContainerDied","Data":"cef0c400f58c0396504168983fa5c8c3e19bf9d720acbfb6454d7bb1dcd520a2"} Nov 02 01:16:13.892748 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.892728 4427 scope.go:122] "RemoveContainer" containerID="0f33a858f189cfcca7f3b166f59bd76ce8e71300eb2a80245e2f38b9548f3a3f" Nov 02 01:16:13.899760 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.899739 4427 scope.go:122] "RemoveContainer" containerID="bc37e69058349568185f4b1203e846b8c7d725831eaf51d34b2f7d2db466d069" Nov 02 01:16:13.907211 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.907187 4427 scope.go:122] "RemoveContainer" containerID="faaf4b41883cfbc689b20cec311640587bcae25c66591b3b1571086ba983b49c" Nov 02 01:16:13.908495 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.908472 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 01:16:13.914145 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.914126 4427 scope.go:122] "RemoveContainer" containerID="4eab1461e0e4faddd78b5114775fe587b85555eec02233eb20ca1f7e771aad55" Nov 02 01:16:13.916854 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.916833 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 01:16:13.922650 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.922626 4427 scope.go:122] "RemoveContainer" containerID="5a8ea4231e84fc19c98ab495d418e2bdcedb1726c88742c560a480f6b087a515" Nov 02 01:16:13.927152 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.927129 4427 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0472228c-7bb0-46c5-928d-2b0804457bca-secret-grpc-tls\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:16:13.927244 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.927154 4427 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0472228c-7bb0-46c5-928d-2b0804457bca-secret-kube-rbac-proxy\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:16:13.927244 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.927173 4427 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0472228c-7bb0-46c5-928d-2b0804457bca-thanos-prometheus-http-client-file\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:16:13.927244 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.927187 4427 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/0472228c-7bb0-46c5-928d-2b0804457bca-config\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:16:13.927244 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.927201 4427 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0472228c-7bb0-46c5-928d-2b0804457bca-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:16:13.927244 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.927216 4427 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0472228c-7bb0-46c5-928d-2b0804457bca-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:16:13.927244 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.927230 4427 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0472228c-7bb0-46c5-928d-2b0804457bca-web-config\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:16:13.927244 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.927243 4427 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0472228c-7bb0-46c5-928d-2b0804457bca-tls-assets\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:16:13.927561 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.927257 4427 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0472228c-7bb0-46c5-928d-2b0804457bca-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:16:13.927561 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.927274 4427 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0472228c-7bb0-46c5-928d-2b0804457bca-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:16:13.927561 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.927290 4427 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0472228c-7bb0-46c5-928d-2b0804457bca-secret-prometheus-k8s-tls\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:16:13.927561 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.927304 4427 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0472228c-7bb0-46c5-928d-2b0804457bca-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:16:13.927561 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.927319 4427 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0472228c-7bb0-46c5-928d-2b0804457bca-config-out\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:16:13.927561 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.927333 4427 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0472228c-7bb0-46c5-928d-2b0804457bca-configmap-metrics-client-ca\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:16:13.927561 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.927347 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d8mz\" (UniqueName: \"kubernetes.io/projected/0472228c-7bb0-46c5-928d-2b0804457bca-kube-api-access-8d8mz\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:16:13.927561 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.927361 4427 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0472228c-7bb0-46c5-928d-2b0804457bca-secret-metrics-client-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:16:13.927561 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.927405 4427 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0472228c-7bb0-46c5-928d-2b0804457bca-prometheus-trusted-ca-bundle\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:16:13.930402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.930384 4427 scope.go:122] "RemoveContainer" containerID="f67e3fcbc2e5b69aef6d52b0047ea0987adfb86dff9b149f9285095c3ffcab70" Nov 02 01:16:13.936934 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.936911 4427 scope.go:122] "RemoveContainer" containerID="79687feb72bf527960e10d6b14f53e17a970b201597bfb56d7f5288f1f3bbca6" Nov 02 01:16:13.937219 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:16:13.937201 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79687feb72bf527960e10d6b14f53e17a970b201597bfb56d7f5288f1f3bbca6\": container with ID starting with 79687feb72bf527960e10d6b14f53e17a970b201597bfb56d7f5288f1f3bbca6 not found: ID does not exist" containerID="79687feb72bf527960e10d6b14f53e17a970b201597bfb56d7f5288f1f3bbca6" Nov 02 01:16:13.937266 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.937228 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79687feb72bf527960e10d6b14f53e17a970b201597bfb56d7f5288f1f3bbca6"} err="failed to get container status \"79687feb72bf527960e10d6b14f53e17a970b201597bfb56d7f5288f1f3bbca6\": rpc error: code = NotFound desc = could not find container \"79687feb72bf527960e10d6b14f53e17a970b201597bfb56d7f5288f1f3bbca6\": container with ID starting with 79687feb72bf527960e10d6b14f53e17a970b201597bfb56d7f5288f1f3bbca6 not found: ID does not exist" Nov 02 01:16:13.937266 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.937260 4427 scope.go:122] "RemoveContainer" containerID="0f33a858f189cfcca7f3b166f59bd76ce8e71300eb2a80245e2f38b9548f3a3f" Nov 02 01:16:13.937518 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:16:13.937500 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f33a858f189cfcca7f3b166f59bd76ce8e71300eb2a80245e2f38b9548f3a3f\": container with ID starting with 0f33a858f189cfcca7f3b166f59bd76ce8e71300eb2a80245e2f38b9548f3a3f not found: ID does not exist" containerID="0f33a858f189cfcca7f3b166f59bd76ce8e71300eb2a80245e2f38b9548f3a3f" Nov 02 01:16:13.937567 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.937524 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f33a858f189cfcca7f3b166f59bd76ce8e71300eb2a80245e2f38b9548f3a3f"} err="failed to get container status \"0f33a858f189cfcca7f3b166f59bd76ce8e71300eb2a80245e2f38b9548f3a3f\": rpc error: code = NotFound desc = could not find container \"0f33a858f189cfcca7f3b166f59bd76ce8e71300eb2a80245e2f38b9548f3a3f\": container with ID starting with 0f33a858f189cfcca7f3b166f59bd76ce8e71300eb2a80245e2f38b9548f3a3f not found: ID does not exist" Nov 02 01:16:13.937567 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.937537 4427 scope.go:122] "RemoveContainer" containerID="bc37e69058349568185f4b1203e846b8c7d725831eaf51d34b2f7d2db466d069" Nov 02 01:16:13.937746 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:16:13.937727 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc37e69058349568185f4b1203e846b8c7d725831eaf51d34b2f7d2db466d069\": container with ID starting with bc37e69058349568185f4b1203e846b8c7d725831eaf51d34b2f7d2db466d069 not found: ID does not exist" containerID="bc37e69058349568185f4b1203e846b8c7d725831eaf51d34b2f7d2db466d069" Nov 02 01:16:13.937787 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.937754 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc37e69058349568185f4b1203e846b8c7d725831eaf51d34b2f7d2db466d069"} err="failed to get container status \"bc37e69058349568185f4b1203e846b8c7d725831eaf51d34b2f7d2db466d069\": rpc error: code = NotFound desc = could not find container \"bc37e69058349568185f4b1203e846b8c7d725831eaf51d34b2f7d2db466d069\": container with ID starting with bc37e69058349568185f4b1203e846b8c7d725831eaf51d34b2f7d2db466d069 not found: ID does not exist" Nov 02 01:16:13.937787 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.937770 4427 scope.go:122] "RemoveContainer" containerID="faaf4b41883cfbc689b20cec311640587bcae25c66591b3b1571086ba983b49c" Nov 02 01:16:13.938002 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:16:13.937987 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faaf4b41883cfbc689b20cec311640587bcae25c66591b3b1571086ba983b49c\": container with ID starting with faaf4b41883cfbc689b20cec311640587bcae25c66591b3b1571086ba983b49c not found: ID does not exist" containerID="faaf4b41883cfbc689b20cec311640587bcae25c66591b3b1571086ba983b49c" Nov 02 01:16:13.938043 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.938006 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faaf4b41883cfbc689b20cec311640587bcae25c66591b3b1571086ba983b49c"} err="failed to get container status \"faaf4b41883cfbc689b20cec311640587bcae25c66591b3b1571086ba983b49c\": rpc error: code = NotFound desc = could not find container \"faaf4b41883cfbc689b20cec311640587bcae25c66591b3b1571086ba983b49c\": container with ID starting with faaf4b41883cfbc689b20cec311640587bcae25c66591b3b1571086ba983b49c not found: ID does not exist" Nov 02 01:16:13.938043 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.938018 4427 scope.go:122] "RemoveContainer" containerID="4eab1461e0e4faddd78b5114775fe587b85555eec02233eb20ca1f7e771aad55" Nov 02 01:16:13.938214 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:16:13.938198 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4eab1461e0e4faddd78b5114775fe587b85555eec02233eb20ca1f7e771aad55\": container with ID starting with 4eab1461e0e4faddd78b5114775fe587b85555eec02233eb20ca1f7e771aad55 not found: ID does not exist" containerID="4eab1461e0e4faddd78b5114775fe587b85555eec02233eb20ca1f7e771aad55" Nov 02 01:16:13.938263 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.938220 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4eab1461e0e4faddd78b5114775fe587b85555eec02233eb20ca1f7e771aad55"} err="failed to get container status \"4eab1461e0e4faddd78b5114775fe587b85555eec02233eb20ca1f7e771aad55\": rpc error: code = NotFound desc = could not find container \"4eab1461e0e4faddd78b5114775fe587b85555eec02233eb20ca1f7e771aad55\": container with ID starting with 4eab1461e0e4faddd78b5114775fe587b85555eec02233eb20ca1f7e771aad55 not found: ID does not exist" Nov 02 01:16:13.938263 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.938235 4427 scope.go:122] "RemoveContainer" containerID="5a8ea4231e84fc19c98ab495d418e2bdcedb1726c88742c560a480f6b087a515" Nov 02 01:16:13.938533 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:16:13.938513 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a8ea4231e84fc19c98ab495d418e2bdcedb1726c88742c560a480f6b087a515\": container with ID starting with 5a8ea4231e84fc19c98ab495d418e2bdcedb1726c88742c560a480f6b087a515 not found: ID does not exist" containerID="5a8ea4231e84fc19c98ab495d418e2bdcedb1726c88742c560a480f6b087a515" Nov 02 01:16:13.938588 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.938537 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a8ea4231e84fc19c98ab495d418e2bdcedb1726c88742c560a480f6b087a515"} err="failed to get container status \"5a8ea4231e84fc19c98ab495d418e2bdcedb1726c88742c560a480f6b087a515\": rpc error: code = NotFound desc = could not find container \"5a8ea4231e84fc19c98ab495d418e2bdcedb1726c88742c560a480f6b087a515\": container with ID starting with 5a8ea4231e84fc19c98ab495d418e2bdcedb1726c88742c560a480f6b087a515 not found: ID does not exist" Nov 02 01:16:13.938588 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.938551 4427 scope.go:122] "RemoveContainer" containerID="f67e3fcbc2e5b69aef6d52b0047ea0987adfb86dff9b149f9285095c3ffcab70" Nov 02 01:16:13.938935 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:16:13.938911 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f67e3fcbc2e5b69aef6d52b0047ea0987adfb86dff9b149f9285095c3ffcab70\": container with ID starting with f67e3fcbc2e5b69aef6d52b0047ea0987adfb86dff9b149f9285095c3ffcab70 not found: ID does not exist" containerID="f67e3fcbc2e5b69aef6d52b0047ea0987adfb86dff9b149f9285095c3ffcab70" Nov 02 01:16:13.938996 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.938942 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f67e3fcbc2e5b69aef6d52b0047ea0987adfb86dff9b149f9285095c3ffcab70"} err="failed to get container status \"f67e3fcbc2e5b69aef6d52b0047ea0987adfb86dff9b149f9285095c3ffcab70\": rpc error: code = NotFound desc = could not find container \"f67e3fcbc2e5b69aef6d52b0047ea0987adfb86dff9b149f9285095c3ffcab70\": container with ID starting with f67e3fcbc2e5b69aef6d52b0047ea0987adfb86dff9b149f9285095c3ffcab70 not found: ID does not exist" Nov 02 01:16:13.938996 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.938956 4427 scope.go:122] "RemoveContainer" containerID="79687feb72bf527960e10d6b14f53e17a970b201597bfb56d7f5288f1f3bbca6" Nov 02 01:16:13.939199 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.939182 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79687feb72bf527960e10d6b14f53e17a970b201597bfb56d7f5288f1f3bbca6"} err="failed to get container status \"79687feb72bf527960e10d6b14f53e17a970b201597bfb56d7f5288f1f3bbca6\": rpc error: code = NotFound desc = could not find container \"79687feb72bf527960e10d6b14f53e17a970b201597bfb56d7f5288f1f3bbca6\": container with ID starting with 79687feb72bf527960e10d6b14f53e17a970b201597bfb56d7f5288f1f3bbca6 not found: ID does not exist" Nov 02 01:16:13.939237 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.939200 4427 scope.go:122] "RemoveContainer" containerID="0f33a858f189cfcca7f3b166f59bd76ce8e71300eb2a80245e2f38b9548f3a3f" Nov 02 01:16:13.939425 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.939409 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f33a858f189cfcca7f3b166f59bd76ce8e71300eb2a80245e2f38b9548f3a3f"} err="failed to get container status \"0f33a858f189cfcca7f3b166f59bd76ce8e71300eb2a80245e2f38b9548f3a3f\": rpc error: code = NotFound desc = could not find container \"0f33a858f189cfcca7f3b166f59bd76ce8e71300eb2a80245e2f38b9548f3a3f\": container with ID starting with 0f33a858f189cfcca7f3b166f59bd76ce8e71300eb2a80245e2f38b9548f3a3f not found: ID does not exist" Nov 02 01:16:13.939471 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.939426 4427 scope.go:122] "RemoveContainer" containerID="bc37e69058349568185f4b1203e846b8c7d725831eaf51d34b2f7d2db466d069" Nov 02 01:16:13.939642 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.939624 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc37e69058349568185f4b1203e846b8c7d725831eaf51d34b2f7d2db466d069"} err="failed to get container status \"bc37e69058349568185f4b1203e846b8c7d725831eaf51d34b2f7d2db466d069\": rpc error: code = NotFound desc = could not find container \"bc37e69058349568185f4b1203e846b8c7d725831eaf51d34b2f7d2db466d069\": container with ID starting with bc37e69058349568185f4b1203e846b8c7d725831eaf51d34b2f7d2db466d069 not found: ID does not exist" Nov 02 01:16:13.939684 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.939643 4427 scope.go:122] "RemoveContainer" containerID="faaf4b41883cfbc689b20cec311640587bcae25c66591b3b1571086ba983b49c" Nov 02 01:16:13.939894 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.939877 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faaf4b41883cfbc689b20cec311640587bcae25c66591b3b1571086ba983b49c"} err="failed to get container status \"faaf4b41883cfbc689b20cec311640587bcae25c66591b3b1571086ba983b49c\": rpc error: code = NotFound desc = could not find container \"faaf4b41883cfbc689b20cec311640587bcae25c66591b3b1571086ba983b49c\": container with ID starting with faaf4b41883cfbc689b20cec311640587bcae25c66591b3b1571086ba983b49c not found: ID does not exist" Nov 02 01:16:13.939938 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.939894 4427 scope.go:122] "RemoveContainer" containerID="4eab1461e0e4faddd78b5114775fe587b85555eec02233eb20ca1f7e771aad55" Nov 02 01:16:13.940137 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.940117 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4eab1461e0e4faddd78b5114775fe587b85555eec02233eb20ca1f7e771aad55"} err="failed to get container status \"4eab1461e0e4faddd78b5114775fe587b85555eec02233eb20ca1f7e771aad55\": rpc error: code = NotFound desc = could not find container \"4eab1461e0e4faddd78b5114775fe587b85555eec02233eb20ca1f7e771aad55\": container with ID starting with 4eab1461e0e4faddd78b5114775fe587b85555eec02233eb20ca1f7e771aad55 not found: ID does not exist" Nov 02 01:16:13.940184 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.940138 4427 scope.go:122] "RemoveContainer" containerID="5a8ea4231e84fc19c98ab495d418e2bdcedb1726c88742c560a480f6b087a515" Nov 02 01:16:13.940353 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.940333 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a8ea4231e84fc19c98ab495d418e2bdcedb1726c88742c560a480f6b087a515"} err="failed to get container status \"5a8ea4231e84fc19c98ab495d418e2bdcedb1726c88742c560a480f6b087a515\": rpc error: code = NotFound desc = could not find container \"5a8ea4231e84fc19c98ab495d418e2bdcedb1726c88742c560a480f6b087a515\": container with ID starting with 5a8ea4231e84fc19c98ab495d418e2bdcedb1726c88742c560a480f6b087a515 not found: ID does not exist" Nov 02 01:16:13.940353 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.940350 4427 scope.go:122] "RemoveContainer" containerID="f67e3fcbc2e5b69aef6d52b0047ea0987adfb86dff9b149f9285095c3ffcab70" Nov 02 01:16:13.940586 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.940566 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f67e3fcbc2e5b69aef6d52b0047ea0987adfb86dff9b149f9285095c3ffcab70"} err="failed to get container status \"f67e3fcbc2e5b69aef6d52b0047ea0987adfb86dff9b149f9285095c3ffcab70\": rpc error: code = NotFound desc = could not find container \"f67e3fcbc2e5b69aef6d52b0047ea0987adfb86dff9b149f9285095c3ffcab70\": container with ID starting with f67e3fcbc2e5b69aef6d52b0047ea0987adfb86dff9b149f9285095c3ffcab70 not found: ID does not exist" Nov 02 01:16:13.940633 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.940586 4427 scope.go:122] "RemoveContainer" containerID="79687feb72bf527960e10d6b14f53e17a970b201597bfb56d7f5288f1f3bbca6" Nov 02 01:16:13.940799 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.940780 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79687feb72bf527960e10d6b14f53e17a970b201597bfb56d7f5288f1f3bbca6"} err="failed to get container status \"79687feb72bf527960e10d6b14f53e17a970b201597bfb56d7f5288f1f3bbca6\": rpc error: code = NotFound desc = could not find container \"79687feb72bf527960e10d6b14f53e17a970b201597bfb56d7f5288f1f3bbca6\": container with ID starting with 79687feb72bf527960e10d6b14f53e17a970b201597bfb56d7f5288f1f3bbca6 not found: ID does not exist" Nov 02 01:16:13.940871 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.940801 4427 scope.go:122] "RemoveContainer" containerID="0f33a858f189cfcca7f3b166f59bd76ce8e71300eb2a80245e2f38b9548f3a3f" Nov 02 01:16:13.941017 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.941001 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f33a858f189cfcca7f3b166f59bd76ce8e71300eb2a80245e2f38b9548f3a3f"} err="failed to get container status \"0f33a858f189cfcca7f3b166f59bd76ce8e71300eb2a80245e2f38b9548f3a3f\": rpc error: code = NotFound desc = could not find container \"0f33a858f189cfcca7f3b166f59bd76ce8e71300eb2a80245e2f38b9548f3a3f\": container with ID starting with 0f33a858f189cfcca7f3b166f59bd76ce8e71300eb2a80245e2f38b9548f3a3f not found: ID does not exist" Nov 02 01:16:13.941068 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.941016 4427 scope.go:122] "RemoveContainer" containerID="bc37e69058349568185f4b1203e846b8c7d725831eaf51d34b2f7d2db466d069" Nov 02 01:16:13.941203 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.941181 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc37e69058349568185f4b1203e846b8c7d725831eaf51d34b2f7d2db466d069"} err="failed to get container status \"bc37e69058349568185f4b1203e846b8c7d725831eaf51d34b2f7d2db466d069\": rpc error: code = NotFound desc = could not find container \"bc37e69058349568185f4b1203e846b8c7d725831eaf51d34b2f7d2db466d069\": container with ID starting with bc37e69058349568185f4b1203e846b8c7d725831eaf51d34b2f7d2db466d069 not found: ID does not exist" Nov 02 01:16:13.941255 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.941205 4427 scope.go:122] "RemoveContainer" containerID="faaf4b41883cfbc689b20cec311640587bcae25c66591b3b1571086ba983b49c" Nov 02 01:16:13.941474 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.941457 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faaf4b41883cfbc689b20cec311640587bcae25c66591b3b1571086ba983b49c"} err="failed to get container status \"faaf4b41883cfbc689b20cec311640587bcae25c66591b3b1571086ba983b49c\": rpc error: code = NotFound desc = could not find container \"faaf4b41883cfbc689b20cec311640587bcae25c66591b3b1571086ba983b49c\": container with ID starting with faaf4b41883cfbc689b20cec311640587bcae25c66591b3b1571086ba983b49c not found: ID does not exist" Nov 02 01:16:13.941474 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.941473 4427 scope.go:122] "RemoveContainer" containerID="4eab1461e0e4faddd78b5114775fe587b85555eec02233eb20ca1f7e771aad55" Nov 02 01:16:13.941720 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.941695 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4eab1461e0e4faddd78b5114775fe587b85555eec02233eb20ca1f7e771aad55"} err="failed to get container status \"4eab1461e0e4faddd78b5114775fe587b85555eec02233eb20ca1f7e771aad55\": rpc error: code = NotFound desc = could not find container \"4eab1461e0e4faddd78b5114775fe587b85555eec02233eb20ca1f7e771aad55\": container with ID starting with 4eab1461e0e4faddd78b5114775fe587b85555eec02233eb20ca1f7e771aad55 not found: ID does not exist" Nov 02 01:16:13.941720 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.941719 4427 scope.go:122] "RemoveContainer" containerID="5a8ea4231e84fc19c98ab495d418e2bdcedb1726c88742c560a480f6b087a515" Nov 02 01:16:13.941956 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.941934 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a8ea4231e84fc19c98ab495d418e2bdcedb1726c88742c560a480f6b087a515"} err="failed to get container status \"5a8ea4231e84fc19c98ab495d418e2bdcedb1726c88742c560a480f6b087a515\": rpc error: code = NotFound desc = could not find container \"5a8ea4231e84fc19c98ab495d418e2bdcedb1726c88742c560a480f6b087a515\": container with ID starting with 5a8ea4231e84fc19c98ab495d418e2bdcedb1726c88742c560a480f6b087a515 not found: ID does not exist" Nov 02 01:16:13.942028 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.941957 4427 scope.go:122] "RemoveContainer" containerID="f67e3fcbc2e5b69aef6d52b0047ea0987adfb86dff9b149f9285095c3ffcab70" Nov 02 01:16:13.942175 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.942151 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f67e3fcbc2e5b69aef6d52b0047ea0987adfb86dff9b149f9285095c3ffcab70"} err="failed to get container status \"f67e3fcbc2e5b69aef6d52b0047ea0987adfb86dff9b149f9285095c3ffcab70\": rpc error: code = NotFound desc = could not find container \"f67e3fcbc2e5b69aef6d52b0047ea0987adfb86dff9b149f9285095c3ffcab70\": container with ID starting with f67e3fcbc2e5b69aef6d52b0047ea0987adfb86dff9b149f9285095c3ffcab70 not found: ID does not exist" Nov 02 01:16:13.942175 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.942172 4427 scope.go:122] "RemoveContainer" containerID="79687feb72bf527960e10d6b14f53e17a970b201597bfb56d7f5288f1f3bbca6" Nov 02 01:16:13.942436 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.942359 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79687feb72bf527960e10d6b14f53e17a970b201597bfb56d7f5288f1f3bbca6"} err="failed to get container status \"79687feb72bf527960e10d6b14f53e17a970b201597bfb56d7f5288f1f3bbca6\": rpc error: code = NotFound desc = could not find container \"79687feb72bf527960e10d6b14f53e17a970b201597bfb56d7f5288f1f3bbca6\": container with ID starting with 79687feb72bf527960e10d6b14f53e17a970b201597bfb56d7f5288f1f3bbca6 not found: ID does not exist" Nov 02 01:16:13.942498 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.942438 4427 scope.go:122] "RemoveContainer" containerID="0f33a858f189cfcca7f3b166f59bd76ce8e71300eb2a80245e2f38b9548f3a3f" Nov 02 01:16:13.942702 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.942684 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f33a858f189cfcca7f3b166f59bd76ce8e71300eb2a80245e2f38b9548f3a3f"} err="failed to get container status \"0f33a858f189cfcca7f3b166f59bd76ce8e71300eb2a80245e2f38b9548f3a3f\": rpc error: code = NotFound desc = could not find container \"0f33a858f189cfcca7f3b166f59bd76ce8e71300eb2a80245e2f38b9548f3a3f\": container with ID starting with 0f33a858f189cfcca7f3b166f59bd76ce8e71300eb2a80245e2f38b9548f3a3f not found: ID does not exist" Nov 02 01:16:13.942757 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.942702 4427 scope.go:122] "RemoveContainer" containerID="bc37e69058349568185f4b1203e846b8c7d725831eaf51d34b2f7d2db466d069" Nov 02 01:16:13.942973 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.942955 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc37e69058349568185f4b1203e846b8c7d725831eaf51d34b2f7d2db466d069"} err="failed to get container status \"bc37e69058349568185f4b1203e846b8c7d725831eaf51d34b2f7d2db466d069\": rpc error: code = NotFound desc = could not find container \"bc37e69058349568185f4b1203e846b8c7d725831eaf51d34b2f7d2db466d069\": container with ID starting with bc37e69058349568185f4b1203e846b8c7d725831eaf51d34b2f7d2db466d069 not found: ID does not exist" Nov 02 01:16:13.943049 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.942975 4427 scope.go:122] "RemoveContainer" containerID="faaf4b41883cfbc689b20cec311640587bcae25c66591b3b1571086ba983b49c" Nov 02 01:16:13.943303 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.943257 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faaf4b41883cfbc689b20cec311640587bcae25c66591b3b1571086ba983b49c"} err="failed to get container status \"faaf4b41883cfbc689b20cec311640587bcae25c66591b3b1571086ba983b49c\": rpc error: code = NotFound desc = could not find container \"faaf4b41883cfbc689b20cec311640587bcae25c66591b3b1571086ba983b49c\": container with ID starting with faaf4b41883cfbc689b20cec311640587bcae25c66591b3b1571086ba983b49c not found: ID does not exist" Nov 02 01:16:13.943303 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.943293 4427 scope.go:122] "RemoveContainer" containerID="4eab1461e0e4faddd78b5114775fe587b85555eec02233eb20ca1f7e771aad55" Nov 02 01:16:13.943561 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.943543 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4eab1461e0e4faddd78b5114775fe587b85555eec02233eb20ca1f7e771aad55"} err="failed to get container status \"4eab1461e0e4faddd78b5114775fe587b85555eec02233eb20ca1f7e771aad55\": rpc error: code = NotFound desc = could not find container \"4eab1461e0e4faddd78b5114775fe587b85555eec02233eb20ca1f7e771aad55\": container with ID starting with 4eab1461e0e4faddd78b5114775fe587b85555eec02233eb20ca1f7e771aad55 not found: ID does not exist" Nov 02 01:16:13.943608 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.943563 4427 scope.go:122] "RemoveContainer" containerID="5a8ea4231e84fc19c98ab495d418e2bdcedb1726c88742c560a480f6b087a515" Nov 02 01:16:13.943834 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.943809 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a8ea4231e84fc19c98ab495d418e2bdcedb1726c88742c560a480f6b087a515"} err="failed to get container status \"5a8ea4231e84fc19c98ab495d418e2bdcedb1726c88742c560a480f6b087a515\": rpc error: code = NotFound desc = could not find container \"5a8ea4231e84fc19c98ab495d418e2bdcedb1726c88742c560a480f6b087a515\": container with ID starting with 5a8ea4231e84fc19c98ab495d418e2bdcedb1726c88742c560a480f6b087a515 not found: ID does not exist" Nov 02 01:16:13.943834 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.943829 4427 scope.go:122] "RemoveContainer" containerID="f67e3fcbc2e5b69aef6d52b0047ea0987adfb86dff9b149f9285095c3ffcab70" Nov 02 01:16:13.944059 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.944042 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f67e3fcbc2e5b69aef6d52b0047ea0987adfb86dff9b149f9285095c3ffcab70"} err="failed to get container status \"f67e3fcbc2e5b69aef6d52b0047ea0987adfb86dff9b149f9285095c3ffcab70\": rpc error: code = NotFound desc = could not find container \"f67e3fcbc2e5b69aef6d52b0047ea0987adfb86dff9b149f9285095c3ffcab70\": container with ID starting with f67e3fcbc2e5b69aef6d52b0047ea0987adfb86dff9b149f9285095c3ffcab70 not found: ID does not exist" Nov 02 01:16:13.944110 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.944060 4427 scope.go:122] "RemoveContainer" containerID="79687feb72bf527960e10d6b14f53e17a970b201597bfb56d7f5288f1f3bbca6" Nov 02 01:16:13.944244 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.944225 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79687feb72bf527960e10d6b14f53e17a970b201597bfb56d7f5288f1f3bbca6"} err="failed to get container status \"79687feb72bf527960e10d6b14f53e17a970b201597bfb56d7f5288f1f3bbca6\": rpc error: code = NotFound desc = could not find container \"79687feb72bf527960e10d6b14f53e17a970b201597bfb56d7f5288f1f3bbca6\": container with ID starting with 79687feb72bf527960e10d6b14f53e17a970b201597bfb56d7f5288f1f3bbca6 not found: ID does not exist" Nov 02 01:16:13.944295 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.944242 4427 scope.go:122] "RemoveContainer" containerID="0f33a858f189cfcca7f3b166f59bd76ce8e71300eb2a80245e2f38b9548f3a3f" Nov 02 01:16:13.944492 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.944477 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f33a858f189cfcca7f3b166f59bd76ce8e71300eb2a80245e2f38b9548f3a3f"} err="failed to get container status \"0f33a858f189cfcca7f3b166f59bd76ce8e71300eb2a80245e2f38b9548f3a3f\": rpc error: code = NotFound desc = could not find container \"0f33a858f189cfcca7f3b166f59bd76ce8e71300eb2a80245e2f38b9548f3a3f\": container with ID starting with 0f33a858f189cfcca7f3b166f59bd76ce8e71300eb2a80245e2f38b9548f3a3f not found: ID does not exist" Nov 02 01:16:13.944538 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.944493 4427 scope.go:122] "RemoveContainer" containerID="bc37e69058349568185f4b1203e846b8c7d725831eaf51d34b2f7d2db466d069" Nov 02 01:16:13.944690 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.944673 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc37e69058349568185f4b1203e846b8c7d725831eaf51d34b2f7d2db466d069"} err="failed to get container status \"bc37e69058349568185f4b1203e846b8c7d725831eaf51d34b2f7d2db466d069\": rpc error: code = NotFound desc = could not find container \"bc37e69058349568185f4b1203e846b8c7d725831eaf51d34b2f7d2db466d069\": container with ID starting with bc37e69058349568185f4b1203e846b8c7d725831eaf51d34b2f7d2db466d069 not found: ID does not exist" Nov 02 01:16:13.944760 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.944692 4427 scope.go:122] "RemoveContainer" containerID="faaf4b41883cfbc689b20cec311640587bcae25c66591b3b1571086ba983b49c" Nov 02 01:16:13.944902 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.944887 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faaf4b41883cfbc689b20cec311640587bcae25c66591b3b1571086ba983b49c"} err="failed to get container status \"faaf4b41883cfbc689b20cec311640587bcae25c66591b3b1571086ba983b49c\": rpc error: code = NotFound desc = could not find container \"faaf4b41883cfbc689b20cec311640587bcae25c66591b3b1571086ba983b49c\": container with ID starting with faaf4b41883cfbc689b20cec311640587bcae25c66591b3b1571086ba983b49c not found: ID does not exist" Nov 02 01:16:13.944963 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.944903 4427 scope.go:122] "RemoveContainer" containerID="4eab1461e0e4faddd78b5114775fe587b85555eec02233eb20ca1f7e771aad55" Nov 02 01:16:13.945099 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.945077 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4eab1461e0e4faddd78b5114775fe587b85555eec02233eb20ca1f7e771aad55"} err="failed to get container status \"4eab1461e0e4faddd78b5114775fe587b85555eec02233eb20ca1f7e771aad55\": rpc error: code = NotFound desc = could not find container \"4eab1461e0e4faddd78b5114775fe587b85555eec02233eb20ca1f7e771aad55\": container with ID starting with 4eab1461e0e4faddd78b5114775fe587b85555eec02233eb20ca1f7e771aad55 not found: ID does not exist" Nov 02 01:16:13.945143 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.945099 4427 scope.go:122] "RemoveContainer" containerID="5a8ea4231e84fc19c98ab495d418e2bdcedb1726c88742c560a480f6b087a515" Nov 02 01:16:13.945316 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.945301 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a8ea4231e84fc19c98ab495d418e2bdcedb1726c88742c560a480f6b087a515"} err="failed to get container status \"5a8ea4231e84fc19c98ab495d418e2bdcedb1726c88742c560a480f6b087a515\": rpc error: code = NotFound desc = could not find container \"5a8ea4231e84fc19c98ab495d418e2bdcedb1726c88742c560a480f6b087a515\": container with ID starting with 5a8ea4231e84fc19c98ab495d418e2bdcedb1726c88742c560a480f6b087a515 not found: ID does not exist" Nov 02 01:16:13.945361 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.945317 4427 scope.go:122] "RemoveContainer" containerID="f67e3fcbc2e5b69aef6d52b0047ea0987adfb86dff9b149f9285095c3ffcab70" Nov 02 01:16:13.945550 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.945520 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f67e3fcbc2e5b69aef6d52b0047ea0987adfb86dff9b149f9285095c3ffcab70"} err="failed to get container status \"f67e3fcbc2e5b69aef6d52b0047ea0987adfb86dff9b149f9285095c3ffcab70\": rpc error: code = NotFound desc = could not find container \"f67e3fcbc2e5b69aef6d52b0047ea0987adfb86dff9b149f9285095c3ffcab70\": container with ID starting with f67e3fcbc2e5b69aef6d52b0047ea0987adfb86dff9b149f9285095c3ffcab70 not found: ID does not exist" Nov 02 01:16:13.945550 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.945548 4427 scope.go:122] "RemoveContainer" containerID="79687feb72bf527960e10d6b14f53e17a970b201597bfb56d7f5288f1f3bbca6" Nov 02 01:16:13.945908 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.945841 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79687feb72bf527960e10d6b14f53e17a970b201597bfb56d7f5288f1f3bbca6"} err="failed to get container status \"79687feb72bf527960e10d6b14f53e17a970b201597bfb56d7f5288f1f3bbca6\": rpc error: code = NotFound desc = could not find container \"79687feb72bf527960e10d6b14f53e17a970b201597bfb56d7f5288f1f3bbca6\": container with ID starting with 79687feb72bf527960e10d6b14f53e17a970b201597bfb56d7f5288f1f3bbca6 not found: ID does not exist" Nov 02 01:16:13.946009 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.945908 4427 scope.go:122] "RemoveContainer" containerID="0f33a858f189cfcca7f3b166f59bd76ce8e71300eb2a80245e2f38b9548f3a3f" Nov 02 01:16:13.946215 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.946194 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f33a858f189cfcca7f3b166f59bd76ce8e71300eb2a80245e2f38b9548f3a3f"} err="failed to get container status \"0f33a858f189cfcca7f3b166f59bd76ce8e71300eb2a80245e2f38b9548f3a3f\": rpc error: code = NotFound desc = could not find container \"0f33a858f189cfcca7f3b166f59bd76ce8e71300eb2a80245e2f38b9548f3a3f\": container with ID starting with 0f33a858f189cfcca7f3b166f59bd76ce8e71300eb2a80245e2f38b9548f3a3f not found: ID does not exist" Nov 02 01:16:13.946310 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.946216 4427 scope.go:122] "RemoveContainer" containerID="bc37e69058349568185f4b1203e846b8c7d725831eaf51d34b2f7d2db466d069" Nov 02 01:16:13.946529 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.946504 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc37e69058349568185f4b1203e846b8c7d725831eaf51d34b2f7d2db466d069"} err="failed to get container status \"bc37e69058349568185f4b1203e846b8c7d725831eaf51d34b2f7d2db466d069\": rpc error: code = NotFound desc = could not find container \"bc37e69058349568185f4b1203e846b8c7d725831eaf51d34b2f7d2db466d069\": container with ID starting with bc37e69058349568185f4b1203e846b8c7d725831eaf51d34b2f7d2db466d069 not found: ID does not exist" Nov 02 01:16:13.946529 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.946528 4427 scope.go:122] "RemoveContainer" containerID="faaf4b41883cfbc689b20cec311640587bcae25c66591b3b1571086ba983b49c" Nov 02 01:16:13.946837 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.946818 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faaf4b41883cfbc689b20cec311640587bcae25c66591b3b1571086ba983b49c"} err="failed to get container status \"faaf4b41883cfbc689b20cec311640587bcae25c66591b3b1571086ba983b49c\": rpc error: code = NotFound desc = could not find container \"faaf4b41883cfbc689b20cec311640587bcae25c66591b3b1571086ba983b49c\": container with ID starting with faaf4b41883cfbc689b20cec311640587bcae25c66591b3b1571086ba983b49c not found: ID does not exist" Nov 02 01:16:13.946837 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.946838 4427 scope.go:122] "RemoveContainer" containerID="4eab1461e0e4faddd78b5114775fe587b85555eec02233eb20ca1f7e771aad55" Nov 02 01:16:13.947137 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.947112 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4eab1461e0e4faddd78b5114775fe587b85555eec02233eb20ca1f7e771aad55"} err="failed to get container status \"4eab1461e0e4faddd78b5114775fe587b85555eec02233eb20ca1f7e771aad55\": rpc error: code = NotFound desc = could not find container \"4eab1461e0e4faddd78b5114775fe587b85555eec02233eb20ca1f7e771aad55\": container with ID starting with 4eab1461e0e4faddd78b5114775fe587b85555eec02233eb20ca1f7e771aad55 not found: ID does not exist" Nov 02 01:16:13.947137 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.947137 4427 scope.go:122] "RemoveContainer" containerID="5a8ea4231e84fc19c98ab495d418e2bdcedb1726c88742c560a480f6b087a515" Nov 02 01:16:13.947288 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.947241 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 01:16:13.948095 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.947955 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a8ea4231e84fc19c98ab495d418e2bdcedb1726c88742c560a480f6b087a515"} err="failed to get container status \"5a8ea4231e84fc19c98ab495d418e2bdcedb1726c88742c560a480f6b087a515\": rpc error: code = NotFound desc = could not find container \"5a8ea4231e84fc19c98ab495d418e2bdcedb1726c88742c560a480f6b087a515\": container with ID starting with 5a8ea4231e84fc19c98ab495d418e2bdcedb1726c88742c560a480f6b087a515 not found: ID does not exist" Nov 02 01:16:13.948095 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.948002 4427 scope.go:122] "RemoveContainer" containerID="f67e3fcbc2e5b69aef6d52b0047ea0987adfb86dff9b149f9285095c3ffcab70" Nov 02 01:16:13.951827 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.948555 4427 cpu_manager.go:414] "containerMap: removing container" podUID="0472228c-7bb0-46c5-928d-2b0804457bca" containerName="thanos-sidecar" Nov 02 01:16:13.951827 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.948575 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0472228c-7bb0-46c5-928d-2b0804457bca" containerName="thanos-sidecar" Nov 02 01:16:13.951827 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.948587 4427 cpu_manager.go:414] "containerMap: removing container" podUID="0472228c-7bb0-46c5-928d-2b0804457bca" containerName="kube-rbac-proxy" Nov 02 01:16:13.951827 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.948595 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0472228c-7bb0-46c5-928d-2b0804457bca" containerName="kube-rbac-proxy" Nov 02 01:16:13.951827 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.948617 4427 cpu_manager.go:414] "containerMap: removing container" podUID="0472228c-7bb0-46c5-928d-2b0804457bca" containerName="init-config-reloader" Nov 02 01:16:13.951827 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.948625 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0472228c-7bb0-46c5-928d-2b0804457bca" containerName="init-config-reloader" Nov 02 01:16:13.951827 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.948649 4427 cpu_manager.go:414] "containerMap: removing container" podUID="0472228c-7bb0-46c5-928d-2b0804457bca" containerName="config-reloader" Nov 02 01:16:13.951827 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.948656 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0472228c-7bb0-46c5-928d-2b0804457bca" containerName="config-reloader" Nov 02 01:16:13.951827 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.948672 4427 cpu_manager.go:414] "containerMap: removing container" podUID="0472228c-7bb0-46c5-928d-2b0804457bca" containerName="kube-rbac-proxy-thanos" Nov 02 01:16:13.951827 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.948679 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0472228c-7bb0-46c5-928d-2b0804457bca" containerName="kube-rbac-proxy-thanos" Nov 02 01:16:13.951827 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.948712 4427 cpu_manager.go:414] "containerMap: removing container" podUID="0472228c-7bb0-46c5-928d-2b0804457bca" containerName="kube-rbac-proxy-web" Nov 02 01:16:13.951827 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.948719 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0472228c-7bb0-46c5-928d-2b0804457bca" containerName="kube-rbac-proxy-web" Nov 02 01:16:13.951827 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.948738 4427 cpu_manager.go:414] "containerMap: removing container" podUID="0472228c-7bb0-46c5-928d-2b0804457bca" containerName="prometheus" Nov 02 01:16:13.951827 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.948745 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0472228c-7bb0-46c5-928d-2b0804457bca" containerName="prometheus" Nov 02 01:16:13.951827 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.948856 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="0472228c-7bb0-46c5-928d-2b0804457bca" containerName="kube-rbac-proxy-web" Nov 02 01:16:13.951827 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.948867 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="0472228c-7bb0-46c5-928d-2b0804457bca" containerName="prometheus" Nov 02 01:16:13.951827 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.948879 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="0472228c-7bb0-46c5-928d-2b0804457bca" containerName="thanos-sidecar" Nov 02 01:16:13.951827 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.948894 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="0472228c-7bb0-46c5-928d-2b0804457bca" containerName="kube-rbac-proxy" Nov 02 01:16:13.951827 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.948905 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="0472228c-7bb0-46c5-928d-2b0804457bca" containerName="config-reloader" Nov 02 01:16:13.951827 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.948912 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="0472228c-7bb0-46c5-928d-2b0804457bca" containerName="kube-rbac-proxy-thanos" Nov 02 01:16:13.954893 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.954862 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f67e3fcbc2e5b69aef6d52b0047ea0987adfb86dff9b149f9285095c3ffcab70"} err="failed to get container status \"f67e3fcbc2e5b69aef6d52b0047ea0987adfb86dff9b149f9285095c3ffcab70\": rpc error: code = NotFound desc = could not find container \"f67e3fcbc2e5b69aef6d52b0047ea0987adfb86dff9b149f9285095c3ffcab70\": container with ID starting with f67e3fcbc2e5b69aef6d52b0047ea0987adfb86dff9b149f9285095c3ffcab70 not found: ID does not exist" Nov 02 01:16:13.956068 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.956042 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:13.961116 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.961095 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 02 01:16:13.961116 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.961112 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 02 01:16:13.961285 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.961139 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 02 01:16:13.961285 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.961169 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 02 01:16:13.961285 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.961186 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 02 01:16:13.961285 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.961200 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 02 01:16:13.961285 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.961210 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 02 01:16:13.961285 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.961212 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-26laidsh9cjjh\"" Nov 02 01:16:13.961285 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.961261 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 02 01:16:13.961660 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.961494 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 02 01:16:13.961925 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.961904 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 02 01:16:13.961925 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.961906 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-49d8x\"" Nov 02 01:16:13.962068 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.961948 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 02 01:16:13.963111 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.963067 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 02 01:16:13.963111 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.963090 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 02 01:16:13.969216 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.969142 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 02 01:16:13.970331 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.969432 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 02 01:16:13.971041 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:13.971013 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 01:16:14.028368 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:14.028337 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a776f00-fcf7-4609-a76d-5527848f1a13-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2a776f00-fcf7-4609-a76d-5527848f1a13\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:14.028544 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:14.028392 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2a776f00-fcf7-4609-a76d-5527848f1a13-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2a776f00-fcf7-4609-a76d-5527848f1a13\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:14.028544 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:14.028483 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2a776f00-fcf7-4609-a76d-5527848f1a13-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2a776f00-fcf7-4609-a76d-5527848f1a13\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:14.028544 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:14.028513 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96vkp\" (UniqueName: \"kubernetes.io/projected/2a776f00-fcf7-4609-a76d-5527848f1a13-kube-api-access-96vkp\") pod \"prometheus-k8s-0\" (UID: \"2a776f00-fcf7-4609-a76d-5527848f1a13\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:14.028644 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:14.028542 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2a776f00-fcf7-4609-a76d-5527848f1a13-config-out\") pod \"prometheus-k8s-0\" (UID: \"2a776f00-fcf7-4609-a76d-5527848f1a13\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:14.028644 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:14.028592 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2a776f00-fcf7-4609-a76d-5527848f1a13-config\") pod \"prometheus-k8s-0\" (UID: \"2a776f00-fcf7-4609-a76d-5527848f1a13\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:14.028644 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:14.028627 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2a776f00-fcf7-4609-a76d-5527848f1a13-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2a776f00-fcf7-4609-a76d-5527848f1a13\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:14.028732 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:14.028646 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2a776f00-fcf7-4609-a76d-5527848f1a13-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2a776f00-fcf7-4609-a76d-5527848f1a13\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:14.028732 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:14.028663 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2a776f00-fcf7-4609-a76d-5527848f1a13-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2a776f00-fcf7-4609-a76d-5527848f1a13\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:14.028732 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:14.028682 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2a776f00-fcf7-4609-a76d-5527848f1a13-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2a776f00-fcf7-4609-a76d-5527848f1a13\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:14.028732 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:14.028700 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2a776f00-fcf7-4609-a76d-5527848f1a13-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2a776f00-fcf7-4609-a76d-5527848f1a13\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:14.028848 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:14.028778 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a776f00-fcf7-4609-a76d-5527848f1a13-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2a776f00-fcf7-4609-a76d-5527848f1a13\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:14.028848 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:14.028835 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a776f00-fcf7-4609-a76d-5527848f1a13-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2a776f00-fcf7-4609-a76d-5527848f1a13\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:14.028905 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:14.028856 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2a776f00-fcf7-4609-a76d-5527848f1a13-web-config\") pod \"prometheus-k8s-0\" (UID: \"2a776f00-fcf7-4609-a76d-5527848f1a13\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:14.028905 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:14.028886 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2a776f00-fcf7-4609-a76d-5527848f1a13-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2a776f00-fcf7-4609-a76d-5527848f1a13\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:14.028962 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:14.028946 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a776f00-fcf7-4609-a76d-5527848f1a13-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2a776f00-fcf7-4609-a76d-5527848f1a13\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:14.028994 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:14.028976 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2a776f00-fcf7-4609-a76d-5527848f1a13-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2a776f00-fcf7-4609-a76d-5527848f1a13\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:14.029026 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:14.028991 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2a776f00-fcf7-4609-a76d-5527848f1a13-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2a776f00-fcf7-4609-a76d-5527848f1a13\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:14.029026 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:14.029008 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2a776f00-fcf7-4609-a76d-5527848f1a13-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2a776f00-fcf7-4609-a76d-5527848f1a13\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:14.029103 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:14.029083 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2a776f00-fcf7-4609-a76d-5527848f1a13-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2a776f00-fcf7-4609-a76d-5527848f1a13\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:14.130186 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:14.130149 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2a776f00-fcf7-4609-a76d-5527848f1a13-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2a776f00-fcf7-4609-a76d-5527848f1a13\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:14.130186 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:14.130185 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2a776f00-fcf7-4609-a76d-5527848f1a13-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2a776f00-fcf7-4609-a76d-5527848f1a13\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:14.130421 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:14.130203 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2a776f00-fcf7-4609-a76d-5527848f1a13-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2a776f00-fcf7-4609-a76d-5527848f1a13\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:14.130421 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:14.130245 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2a776f00-fcf7-4609-a76d-5527848f1a13-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2a776f00-fcf7-4609-a76d-5527848f1a13\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:14.130492 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:14.130430 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2a776f00-fcf7-4609-a76d-5527848f1a13-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2a776f00-fcf7-4609-a76d-5527848f1a13\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:14.130492 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:14.130476 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a776f00-fcf7-4609-a76d-5527848f1a13-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2a776f00-fcf7-4609-a76d-5527848f1a13\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:14.130570 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:14.130512 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a776f00-fcf7-4609-a76d-5527848f1a13-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2a776f00-fcf7-4609-a76d-5527848f1a13\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:14.130570 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:14.130542 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2a776f00-fcf7-4609-a76d-5527848f1a13-web-config\") pod \"prometheus-k8s-0\" (UID: \"2a776f00-fcf7-4609-a76d-5527848f1a13\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:14.130673 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:14.130576 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2a776f00-fcf7-4609-a76d-5527848f1a13-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2a776f00-fcf7-4609-a76d-5527848f1a13\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:14.130673 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:14.130605 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a776f00-fcf7-4609-a76d-5527848f1a13-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2a776f00-fcf7-4609-a76d-5527848f1a13\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:14.130673 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:14.130634 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2a776f00-fcf7-4609-a76d-5527848f1a13-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2a776f00-fcf7-4609-a76d-5527848f1a13\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:14.130673 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:14.130658 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2a776f00-fcf7-4609-a76d-5527848f1a13-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2a776f00-fcf7-4609-a76d-5527848f1a13\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:14.130872 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:14.130683 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2a776f00-fcf7-4609-a76d-5527848f1a13-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2a776f00-fcf7-4609-a76d-5527848f1a13\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:14.130872 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:14.130734 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2a776f00-fcf7-4609-a76d-5527848f1a13-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2a776f00-fcf7-4609-a76d-5527848f1a13\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:14.130872 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:14.130761 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a776f00-fcf7-4609-a76d-5527848f1a13-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2a776f00-fcf7-4609-a76d-5527848f1a13\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:14.130872 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:14.130792 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2a776f00-fcf7-4609-a76d-5527848f1a13-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2a776f00-fcf7-4609-a76d-5527848f1a13\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:14.130872 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:14.130820 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2a776f00-fcf7-4609-a76d-5527848f1a13-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2a776f00-fcf7-4609-a76d-5527848f1a13\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:14.130872 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:14.130842 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96vkp\" (UniqueName: \"kubernetes.io/projected/2a776f00-fcf7-4609-a76d-5527848f1a13-kube-api-access-96vkp\") pod \"prometheus-k8s-0\" (UID: \"2a776f00-fcf7-4609-a76d-5527848f1a13\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:14.131192 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:14.130878 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2a776f00-fcf7-4609-a76d-5527848f1a13-config-out\") pod \"prometheus-k8s-0\" (UID: \"2a776f00-fcf7-4609-a76d-5527848f1a13\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:14.131192 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:14.130898 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2a776f00-fcf7-4609-a76d-5527848f1a13-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2a776f00-fcf7-4609-a76d-5527848f1a13\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:14.131192 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:14.130909 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2a776f00-fcf7-4609-a76d-5527848f1a13-config\") pod \"prometheus-k8s-0\" (UID: \"2a776f00-fcf7-4609-a76d-5527848f1a13\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:14.131338 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:14.131236 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a776f00-fcf7-4609-a76d-5527848f1a13-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2a776f00-fcf7-4609-a76d-5527848f1a13\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:14.131741 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:14.131639 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a776f00-fcf7-4609-a76d-5527848f1a13-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2a776f00-fcf7-4609-a76d-5527848f1a13\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:14.132457 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:14.132429 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a776f00-fcf7-4609-a76d-5527848f1a13-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2a776f00-fcf7-4609-a76d-5527848f1a13\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:14.133710 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:14.133613 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2a776f00-fcf7-4609-a76d-5527848f1a13-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2a776f00-fcf7-4609-a76d-5527848f1a13\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:14.134440 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:14.133835 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2a776f00-fcf7-4609-a76d-5527848f1a13-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2a776f00-fcf7-4609-a76d-5527848f1a13\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:14.134440 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:14.133956 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/2a776f00-fcf7-4609-a76d-5527848f1a13-config\") pod \"prometheus-k8s-0\" (UID: \"2a776f00-fcf7-4609-a76d-5527848f1a13\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:14.134440 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:14.134008 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2a776f00-fcf7-4609-a76d-5527848f1a13-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2a776f00-fcf7-4609-a76d-5527848f1a13\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:14.134440 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:14.134305 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2a776f00-fcf7-4609-a76d-5527848f1a13-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2a776f00-fcf7-4609-a76d-5527848f1a13\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:14.134440 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:14.134334 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2a776f00-fcf7-4609-a76d-5527848f1a13-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2a776f00-fcf7-4609-a76d-5527848f1a13\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:14.134440 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:14.130791 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2a776f00-fcf7-4609-a76d-5527848f1a13-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2a776f00-fcf7-4609-a76d-5527848f1a13\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:14.134440 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:14.134366 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2a776f00-fcf7-4609-a76d-5527848f1a13-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2a776f00-fcf7-4609-a76d-5527848f1a13\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:14.134811 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:14.134568 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a776f00-fcf7-4609-a76d-5527848f1a13-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2a776f00-fcf7-4609-a76d-5527848f1a13\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:14.134811 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:14.134790 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2a776f00-fcf7-4609-a76d-5527848f1a13-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2a776f00-fcf7-4609-a76d-5527848f1a13\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:14.135145 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:14.135118 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2a776f00-fcf7-4609-a76d-5527848f1a13-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2a776f00-fcf7-4609-a76d-5527848f1a13\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:14.136399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:14.136359 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2a776f00-fcf7-4609-a76d-5527848f1a13-config-out\") pod \"prometheus-k8s-0\" (UID: \"2a776f00-fcf7-4609-a76d-5527848f1a13\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:14.136503 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:14.136487 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2a776f00-fcf7-4609-a76d-5527848f1a13-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2a776f00-fcf7-4609-a76d-5527848f1a13\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:14.136800 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:14.136783 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2a776f00-fcf7-4609-a76d-5527848f1a13-web-config\") pod \"prometheus-k8s-0\" (UID: \"2a776f00-fcf7-4609-a76d-5527848f1a13\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:14.136902 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:14.136885 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2a776f00-fcf7-4609-a76d-5527848f1a13-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2a776f00-fcf7-4609-a76d-5527848f1a13\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:14.146462 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:14.146441 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96vkp\" (UniqueName: \"kubernetes.io/projected/2a776f00-fcf7-4609-a76d-5527848f1a13-kube-api-access-96vkp\") pod \"prometheus-k8s-0\" (UID: \"2a776f00-fcf7-4609-a76d-5527848f1a13\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:14.282586 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:14.282509 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:14.571170 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:14.571114 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 01:16:14.573536 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:16:14.573509 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a776f00_fcf7_4609_a76d_5527848f1a13.slice/crio-f47e5698881e5e2dfe52122a6b813f6733bf8e803e8b2b29b71c544997459817 WatchSource:0}: Error finding container f47e5698881e5e2dfe52122a6b813f6733bf8e803e8b2b29b71c544997459817: Status 404 returned error can't find the container with id f47e5698881e5e2dfe52122a6b813f6733bf8e803e8b2b29b71c544997459817 Nov 02 01:16:14.890492 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:14.890458 4427 generic.go:356] "Generic (PLEG): container finished" podID="2a776f00-fcf7-4609-a76d-5527848f1a13" containerID="9b7996927445c289c07746e5ec8af8e83f12ff6788946c862f78124eb16d1699" exitCode=0 Nov 02 01:16:14.890700 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:14.890504 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2a776f00-fcf7-4609-a76d-5527848f1a13","Type":"ContainerDied","Data":"9b7996927445c289c07746e5ec8af8e83f12ff6788946c862f78124eb16d1699"} Nov 02 01:16:14.890700 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:14.890568 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2a776f00-fcf7-4609-a76d-5527848f1a13","Type":"ContainerStarted","Data":"f47e5698881e5e2dfe52122a6b813f6733bf8e803e8b2b29b71c544997459817"} Nov 02 01:16:15.532390 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:15.532338 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0472228c-7bb0-46c5-928d-2b0804457bca" path="/var/lib/kubelet/pods/0472228c-7bb0-46c5-928d-2b0804457bca/volumes" Nov 02 01:16:15.898621 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:15.898593 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2a776f00-fcf7-4609-a76d-5527848f1a13","Type":"ContainerStarted","Data":"f76e00e42baa716eabc8045eb93ed18cc3e00bebbe6195bded5056a258395715"} Nov 02 01:16:15.898986 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:15.898654 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2a776f00-fcf7-4609-a76d-5527848f1a13","Type":"ContainerStarted","Data":"049dbc701a456636308cf116e4d4a380f421488be609e8c4948a85bee557edcc"} Nov 02 01:16:15.898986 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:15.898682 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2a776f00-fcf7-4609-a76d-5527848f1a13","Type":"ContainerStarted","Data":"eef3ad03b5eb8a1f79aeb04b1a18c045c8b85a1353d11a5d61e79f93dcda83f4"} Nov 02 01:16:15.898986 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:15.898706 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2a776f00-fcf7-4609-a76d-5527848f1a13","Type":"ContainerStarted","Data":"4ca65773441ab23674483e7bb7fe48ae79c282bb69f75020ded6e214d0851fde"} Nov 02 01:16:15.898986 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:15.898735 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2a776f00-fcf7-4609-a76d-5527848f1a13","Type":"ContainerStarted","Data":"1597b74a4b33462d37894c44b2dfab4c64621896aceb8b584efab7d8be6b8290"} Nov 02 01:16:15.898986 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:15.898774 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2a776f00-fcf7-4609-a76d-5527848f1a13","Type":"ContainerStarted","Data":"1905c8fffae15a0d5b36fc51448aa9add746f4f66a2f05d8b5002eba6f911126"} Nov 02 01:16:15.936248 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:15.936195 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.920503543 podStartE2EDuration="2.936180972s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.677429ms" isStatefulPod=false podCreationTimestamp="2026-11-02 01:16:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:16:15.933758622 +0000 UTC m=+80.554763835" watchObservedRunningTime="2026-11-02 01:16:15.936180972 +0000 UTC m=+80.557186183" Nov 02 01:16:19.282912 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:16:19.282875 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:16:31.580229 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:16:31.580191 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:17:01.580361 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:17:01.580314 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:17:14.283602 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:17:14.283528 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:17:14.298757 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:17:14.298732 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:17:15.126535 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:17:15.126507 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 01:17:31.579478 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:17:31.579443 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:18:01.579688 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:18:01.579649 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:18:31.581034 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:18:31.580996 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:18:55.478593 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:18:55.478561 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2vsqv"] Nov 02 01:18:55.481842 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:18:55.481825 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2vsqv" Nov 02 01:18:55.486747 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:18:55.486721 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 02 01:18:55.486872 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:18:55.486801 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 02 01:18:55.486872 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:18:55.486834 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-v82s5\"" Nov 02 01:18:55.491077 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:18:55.491055 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2vsqv"] Nov 02 01:18:55.609852 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:18:55.609825 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0662aaa7-9454-4f80-86bd-1862f5ae8636-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2vsqv\" (UID: \"0662aaa7-9454-4f80-86bd-1862f5ae8636\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2vsqv" Nov 02 01:18:55.610006 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:18:55.609873 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbmkk\" (UniqueName: \"kubernetes.io/projected/0662aaa7-9454-4f80-86bd-1862f5ae8636-kube-api-access-vbmkk\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2vsqv\" (UID: \"0662aaa7-9454-4f80-86bd-1862f5ae8636\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2vsqv" Nov 02 01:18:55.610006 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:18:55.609895 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0662aaa7-9454-4f80-86bd-1862f5ae8636-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2vsqv\" (UID: \"0662aaa7-9454-4f80-86bd-1862f5ae8636\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2vsqv" Nov 02 01:18:55.710888 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:18:55.710801 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbmkk\" (UniqueName: \"kubernetes.io/projected/0662aaa7-9454-4f80-86bd-1862f5ae8636-kube-api-access-vbmkk\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2vsqv\" (UID: \"0662aaa7-9454-4f80-86bd-1862f5ae8636\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2vsqv" Nov 02 01:18:55.710888 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:18:55.710835 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0662aaa7-9454-4f80-86bd-1862f5ae8636-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2vsqv\" (UID: \"0662aaa7-9454-4f80-86bd-1862f5ae8636\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2vsqv" Nov 02 01:18:55.711131 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:18:55.710901 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0662aaa7-9454-4f80-86bd-1862f5ae8636-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2vsqv\" (UID: \"0662aaa7-9454-4f80-86bd-1862f5ae8636\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2vsqv" Nov 02 01:18:55.711263 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:18:55.711241 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0662aaa7-9454-4f80-86bd-1862f5ae8636-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2vsqv\" (UID: \"0662aaa7-9454-4f80-86bd-1862f5ae8636\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2vsqv" Nov 02 01:18:55.711327 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:18:55.711307 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0662aaa7-9454-4f80-86bd-1862f5ae8636-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2vsqv\" (UID: \"0662aaa7-9454-4f80-86bd-1862f5ae8636\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2vsqv" Nov 02 01:18:55.723893 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:18:55.723867 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbmkk\" (UniqueName: \"kubernetes.io/projected/0662aaa7-9454-4f80-86bd-1862f5ae8636-kube-api-access-vbmkk\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2vsqv\" (UID: \"0662aaa7-9454-4f80-86bd-1862f5ae8636\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2vsqv" Nov 02 01:18:55.798969 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:18:55.798919 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2vsqv" Nov 02 01:18:56.073179 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:18:56.073133 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2vsqv"] Nov 02 01:18:56.075888 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:18:56.075860 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0662aaa7_9454_4f80_86bd_1862f5ae8636.slice/crio-68ef8436535873c63c80964a4060fb0b0b91c16f5240aeb304ad726180a74280 WatchSource:0}: Error finding container 68ef8436535873c63c80964a4060fb0b0b91c16f5240aeb304ad726180a74280: Status 404 returned error can't find the container with id 68ef8436535873c63c80964a4060fb0b0b91c16f5240aeb304ad726180a74280 Nov 02 01:18:56.455659 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:18:56.455620 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2vsqv" event={"ID":"0662aaa7-9454-4f80-86bd-1862f5ae8636","Type":"ContainerStarted","Data":"68ef8436535873c63c80964a4060fb0b0b91c16f5240aeb304ad726180a74280"} Nov 02 01:19:01.581706 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:19:01.581669 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:19:06.498582 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:06.498546 4427 generic.go:356] "Generic (PLEG): container finished" podID="0662aaa7-9454-4f80-86bd-1862f5ae8636" containerID="950beee3d2fa0d3ccce9a8d52332e6c9790a009e57611da6dcb64fb318302caa" exitCode=0 Nov 02 01:19:06.499010 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:06.498630 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2vsqv" event={"ID":"0662aaa7-9454-4f80-86bd-1862f5ae8636","Type":"ContainerDied","Data":"950beee3d2fa0d3ccce9a8d52332e6c9790a009e57611da6dcb64fb318302caa"} Nov 02 01:19:08.508243 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:08.508209 4427 generic.go:356] "Generic (PLEG): container finished" podID="0662aaa7-9454-4f80-86bd-1862f5ae8636" containerID="ffea5430960443d4926377f5f2b6bfb9317981b4a817f3c49af556631948b560" exitCode=0 Nov 02 01:19:08.508670 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:08.508273 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2vsqv" event={"ID":"0662aaa7-9454-4f80-86bd-1862f5ae8636","Type":"ContainerDied","Data":"ffea5430960443d4926377f5f2b6bfb9317981b4a817f3c49af556631948b560"} Nov 02 01:19:20.550662 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:20.550621 4427 generic.go:356] "Generic (PLEG): container finished" podID="0662aaa7-9454-4f80-86bd-1862f5ae8636" containerID="c0d90eeb15d50a554978a340facf5938c878d5b39dabda45cb962d8e229cdc65" exitCode=0 Nov 02 01:19:20.551079 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:20.550666 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2vsqv" event={"ID":"0662aaa7-9454-4f80-86bd-1862f5ae8636","Type":"ContainerDied","Data":"c0d90eeb15d50a554978a340facf5938c878d5b39dabda45cb962d8e229cdc65"} Nov 02 01:19:21.682414 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:21.682367 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2vsqv" Nov 02 01:19:21.725220 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:21.725184 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0662aaa7-9454-4f80-86bd-1862f5ae8636-util\" (UniqueName: \"kubernetes.io/empty-dir/0662aaa7-9454-4f80-86bd-1862f5ae8636-util\") pod \"0662aaa7-9454-4f80-86bd-1862f5ae8636\" (UID: \"0662aaa7-9454-4f80-86bd-1862f5ae8636\") " Nov 02 01:19:21.725430 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:21.725265 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0662aaa7-9454-4f80-86bd-1862f5ae8636-bundle\" (UniqueName: \"kubernetes.io/empty-dir/0662aaa7-9454-4f80-86bd-1862f5ae8636-bundle\") pod \"0662aaa7-9454-4f80-86bd-1862f5ae8636\" (UID: \"0662aaa7-9454-4f80-86bd-1862f5ae8636\") " Nov 02 01:19:21.725430 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:21.725304 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0662aaa7-9454-4f80-86bd-1862f5ae8636-kube-api-access-vbmkk\" (UniqueName: \"kubernetes.io/projected/0662aaa7-9454-4f80-86bd-1862f5ae8636-kube-api-access-vbmkk\") pod \"0662aaa7-9454-4f80-86bd-1862f5ae8636\" (UID: \"0662aaa7-9454-4f80-86bd-1862f5ae8636\") " Nov 02 01:19:21.725806 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:21.725777 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0662aaa7-9454-4f80-86bd-1862f5ae8636-bundle" pod "0662aaa7-9454-4f80-86bd-1862f5ae8636" (UID: "0662aaa7-9454-4f80-86bd-1862f5ae8636"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:19:21.727702 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:21.727672 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0662aaa7-9454-4f80-86bd-1862f5ae8636-kube-api-access-vbmkk" pod "0662aaa7-9454-4f80-86bd-1862f5ae8636" (UID: "0662aaa7-9454-4f80-86bd-1862f5ae8636"). InnerVolumeSpecName "kube-api-access-vbmkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:19:21.729802 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:21.729779 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0662aaa7-9454-4f80-86bd-1862f5ae8636-util" pod "0662aaa7-9454-4f80-86bd-1862f5ae8636" (UID: "0662aaa7-9454-4f80-86bd-1862f5ae8636"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:19:21.826877 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:21.826790 4427 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0662aaa7-9454-4f80-86bd-1862f5ae8636-bundle\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:19:21.826877 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:21.826821 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbmkk\" (UniqueName: \"kubernetes.io/projected/0662aaa7-9454-4f80-86bd-1862f5ae8636-kube-api-access-vbmkk\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:19:21.826877 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:21.826830 4427 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0662aaa7-9454-4f80-86bd-1862f5ae8636-util\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:19:22.559271 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:22.559237 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2vsqv" event={"ID":"0662aaa7-9454-4f80-86bd-1862f5ae8636","Type":"ContainerDied","Data":"68ef8436535873c63c80964a4060fb0b0b91c16f5240aeb304ad726180a74280"} Nov 02 01:19:22.559271 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:22.559276 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2vsqv" Nov 02 01:19:22.559499 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:22.559282 4427 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68ef8436535873c63c80964a4060fb0b0b91c16f5240aeb304ad726180a74280" Nov 02 01:19:31.580194 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:19:31.580153 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:19:38.517774 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:38.517737 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-zxd5z"] Nov 02 01:19:38.518184 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:38.518037 4427 cpu_manager.go:414] "containerMap: removing container" podUID="0662aaa7-9454-4f80-86bd-1862f5ae8636" containerName="util" Nov 02 01:19:38.518184 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:38.518049 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0662aaa7-9454-4f80-86bd-1862f5ae8636" containerName="util" Nov 02 01:19:38.518184 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:38.518060 4427 cpu_manager.go:414] "containerMap: removing container" podUID="0662aaa7-9454-4f80-86bd-1862f5ae8636" containerName="extract" Nov 02 01:19:38.518184 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:38.518065 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0662aaa7-9454-4f80-86bd-1862f5ae8636" containerName="extract" Nov 02 01:19:38.518184 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:38.518091 4427 cpu_manager.go:414] "containerMap: removing container" podUID="0662aaa7-9454-4f80-86bd-1862f5ae8636" containerName="pull" Nov 02 01:19:38.518184 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:38.518098 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0662aaa7-9454-4f80-86bd-1862f5ae8636" containerName="pull" Nov 02 01:19:38.518184 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:38.518146 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="0662aaa7-9454-4f80-86bd-1862f5ae8636" containerName="extract" Nov 02 01:19:38.521075 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:38.521058 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-zxd5z" Nov 02 01:19:38.553543 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:38.553506 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-zxd5z"] Nov 02 01:19:38.600134 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:38.600099 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 02 01:19:38.600420 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:38.600177 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 02 01:19:38.600420 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:38.600192 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 02 01:19:38.600420 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:38.600308 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-vj7qc\"" Nov 02 01:19:38.600562 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:38.600441 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 02 01:19:38.600562 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:38.600494 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 02 01:19:38.664207 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:38.664170 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2ff1362b-0201-4e45-8aeb-535413957e33-manifests\") pod \"opendatahub-operator-controller-manager-7d6cbb5df9-zxd5z\" (UID: \"2ff1362b-0201-4e45-8aeb-535413957e33\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-zxd5z" Nov 02 01:19:38.664429 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:38.664216 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2ff1362b-0201-4e45-8aeb-535413957e33-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d6cbb5df9-zxd5z\" (UID: \"2ff1362b-0201-4e45-8aeb-535413957e33\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-zxd5z" Nov 02 01:19:38.664429 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:38.664300 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2ff1362b-0201-4e45-8aeb-535413957e33-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d6cbb5df9-zxd5z\" (UID: \"2ff1362b-0201-4e45-8aeb-535413957e33\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-zxd5z" Nov 02 01:19:38.664429 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:38.664394 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2ff1362b-0201-4e45-8aeb-535413957e33-metrics-cert\") pod \"opendatahub-operator-controller-manager-7d6cbb5df9-zxd5z\" (UID: \"2ff1362b-0201-4e45-8aeb-535413957e33\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-zxd5z" Nov 02 01:19:38.664604 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:38.664474 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psv6n\" (UniqueName: \"kubernetes.io/projected/2ff1362b-0201-4e45-8aeb-535413957e33-kube-api-access-psv6n\") pod \"opendatahub-operator-controller-manager-7d6cbb5df9-zxd5z\" (UID: \"2ff1362b-0201-4e45-8aeb-535413957e33\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-zxd5z" Nov 02 01:19:38.664604 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:38.664519 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ff1362b-0201-4e45-8aeb-535413957e33-tmp\") pod \"opendatahub-operator-controller-manager-7d6cbb5df9-zxd5z\" (UID: \"2ff1362b-0201-4e45-8aeb-535413957e33\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-zxd5z" Nov 02 01:19:38.765189 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:38.765158 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2ff1362b-0201-4e45-8aeb-535413957e33-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d6cbb5df9-zxd5z\" (UID: \"2ff1362b-0201-4e45-8aeb-535413957e33\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-zxd5z" Nov 02 01:19:38.765189 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:38.765193 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2ff1362b-0201-4e45-8aeb-535413957e33-metrics-cert\") pod \"opendatahub-operator-controller-manager-7d6cbb5df9-zxd5z\" (UID: \"2ff1362b-0201-4e45-8aeb-535413957e33\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-zxd5z" Nov 02 01:19:38.765455 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:38.765223 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psv6n\" (UniqueName: \"kubernetes.io/projected/2ff1362b-0201-4e45-8aeb-535413957e33-kube-api-access-psv6n\") pod \"opendatahub-operator-controller-manager-7d6cbb5df9-zxd5z\" (UID: \"2ff1362b-0201-4e45-8aeb-535413957e33\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-zxd5z" Nov 02 01:19:38.765455 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:38.765257 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ff1362b-0201-4e45-8aeb-535413957e33-tmp\") pod \"opendatahub-operator-controller-manager-7d6cbb5df9-zxd5z\" (UID: \"2ff1362b-0201-4e45-8aeb-535413957e33\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-zxd5z" Nov 02 01:19:38.765455 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:38.765293 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2ff1362b-0201-4e45-8aeb-535413957e33-manifests\") pod \"opendatahub-operator-controller-manager-7d6cbb5df9-zxd5z\" (UID: \"2ff1362b-0201-4e45-8aeb-535413957e33\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-zxd5z" Nov 02 01:19:38.765455 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:38.765316 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2ff1362b-0201-4e45-8aeb-535413957e33-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d6cbb5df9-zxd5z\" (UID: \"2ff1362b-0201-4e45-8aeb-535413957e33\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-zxd5z" Nov 02 01:19:38.765803 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:38.765773 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2ff1362b-0201-4e45-8aeb-535413957e33-manifests\") pod \"opendatahub-operator-controller-manager-7d6cbb5df9-zxd5z\" (UID: \"2ff1362b-0201-4e45-8aeb-535413957e33\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-zxd5z" Nov 02 01:19:38.768017 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:38.767960 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2ff1362b-0201-4e45-8aeb-535413957e33-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d6cbb5df9-zxd5z\" (UID: \"2ff1362b-0201-4e45-8aeb-535413957e33\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-zxd5z" Nov 02 01:19:38.768105 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:38.768024 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2ff1362b-0201-4e45-8aeb-535413957e33-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d6cbb5df9-zxd5z\" (UID: \"2ff1362b-0201-4e45-8aeb-535413957e33\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-zxd5z" Nov 02 01:19:38.768199 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:38.768181 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ff1362b-0201-4e45-8aeb-535413957e33-tmp\") pod \"opendatahub-operator-controller-manager-7d6cbb5df9-zxd5z\" (UID: \"2ff1362b-0201-4e45-8aeb-535413957e33\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-zxd5z" Nov 02 01:19:38.768415 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:38.768389 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2ff1362b-0201-4e45-8aeb-535413957e33-metrics-cert\") pod \"opendatahub-operator-controller-manager-7d6cbb5df9-zxd5z\" (UID: \"2ff1362b-0201-4e45-8aeb-535413957e33\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-zxd5z" Nov 02 01:19:38.789392 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:38.789352 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psv6n\" (UniqueName: \"kubernetes.io/projected/2ff1362b-0201-4e45-8aeb-535413957e33-kube-api-access-psv6n\") pod \"opendatahub-operator-controller-manager-7d6cbb5df9-zxd5z\" (UID: \"2ff1362b-0201-4e45-8aeb-535413957e33\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-zxd5z" Nov 02 01:19:38.837367 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:38.837326 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-zxd5z" Nov 02 01:19:39.137088 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:39.137036 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-zxd5z"] Nov 02 01:19:39.139358 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:19:39.139332 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ff1362b_0201_4e45_8aeb_535413957e33.slice/crio-46329fbe367af480457e7c6afaaa5566608ba4e5fecf47bf1ee7adc7bdc2c2de WatchSource:0}: Error finding container 46329fbe367af480457e7c6afaaa5566608ba4e5fecf47bf1ee7adc7bdc2c2de: Status 404 returned error can't find the container with id 46329fbe367af480457e7c6afaaa5566608ba4e5fecf47bf1ee7adc7bdc2c2de Nov 02 01:19:39.620313 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:39.620281 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-zxd5z" event={"ID":"2ff1362b-0201-4e45-8aeb-535413957e33","Type":"ContainerStarted","Data":"46329fbe367af480457e7c6afaaa5566608ba4e5fecf47bf1ee7adc7bdc2c2de"} Nov 02 01:19:44.641581 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:44.641546 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-zxd5z" event={"ID":"2ff1362b-0201-4e45-8aeb-535413957e33","Type":"ContainerStarted","Data":"a9d6981c6574a91d5fc488852496294e51609259ee1071957599639af3b2b06b"} Nov 02 01:19:45.646223 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:45.646189 4427 generic.go:356] "Generic (PLEG): container finished" podID="2ff1362b-0201-4e45-8aeb-535413957e33" containerID="a9d6981c6574a91d5fc488852496294e51609259ee1071957599639af3b2b06b" exitCode=0 Nov 02 01:19:45.646626 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:45.646261 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-zxd5z" event={"ID":"2ff1362b-0201-4e45-8aeb-535413957e33","Type":"ContainerDied","Data":"a9d6981c6574a91d5fc488852496294e51609259ee1071957599639af3b2b06b"} Nov 02 01:19:46.651806 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:46.651769 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-zxd5z" event={"ID":"2ff1362b-0201-4e45-8aeb-535413957e33","Type":"ContainerStarted","Data":"c757f2359e1e0a9ee2fcc6c6e0a8e88694d24e93c0f32d7bbb80c99973a0c34b"} Nov 02 01:19:46.652220 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:46.651891 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-zxd5z" Nov 02 01:19:46.711462 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:46.711410 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-zxd5z" podStartSLOduration=2.692573813 podStartE2EDuration="8.711390727s" totalImagesPullingTime="5.583970602s" totalInitContainerRuntime="434.846312ms" isStatefulPod=false podCreationTimestamp="2026-11-02 01:19:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:19:46.708419789 +0000 UTC m=+291.329425000" watchObservedRunningTime="2026-11-02 01:19:46.711390727 +0000 UTC m=+291.332395932" Nov 02 01:19:57.657590 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:57.657555 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6cbb5df9-zxd5z" Nov 02 01:19:58.648142 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:58.648110 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-nddkv"] Nov 02 01:19:58.653662 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:58.653641 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-nddkv" Nov 02 01:19:58.662487 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:58.662460 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 02 01:19:58.663404 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:58.663361 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 02 01:19:58.665160 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:58.665139 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-nddkv"] Nov 02 01:19:58.835235 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:58.835203 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgjgz\" (UniqueName: \"kubernetes.io/projected/4379e4b2-e68b-4ddf-a3bf-9ec404e61f56-kube-api-access-vgjgz\") pod \"cert-manager-webhook-5dbbbf75cc-nddkv\" (UID: \"4379e4b2-e68b-4ddf-a3bf-9ec404e61f56\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-nddkv" Nov 02 01:19:58.835472 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:58.835276 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4379e4b2-e68b-4ddf-a3bf-9ec404e61f56-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-nddkv\" (UID: \"4379e4b2-e68b-4ddf-a3bf-9ec404e61f56\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-nddkv" Nov 02 01:19:58.936627 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:58.936538 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4379e4b2-e68b-4ddf-a3bf-9ec404e61f56-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-nddkv\" (UID: \"4379e4b2-e68b-4ddf-a3bf-9ec404e61f56\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-nddkv" Nov 02 01:19:58.936627 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:58.936600 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgjgz\" (UniqueName: \"kubernetes.io/projected/4379e4b2-e68b-4ddf-a3bf-9ec404e61f56-kube-api-access-vgjgz\") pod \"cert-manager-webhook-5dbbbf75cc-nddkv\" (UID: \"4379e4b2-e68b-4ddf-a3bf-9ec404e61f56\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-nddkv" Nov 02 01:19:58.951888 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:58.951856 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4379e4b2-e68b-4ddf-a3bf-9ec404e61f56-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-nddkv\" (UID: \"4379e4b2-e68b-4ddf-a3bf-9ec404e61f56\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-nddkv" Nov 02 01:19:58.952006 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:58.951992 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgjgz\" (UniqueName: \"kubernetes.io/projected/4379e4b2-e68b-4ddf-a3bf-9ec404e61f56-kube-api-access-vgjgz\") pod \"cert-manager-webhook-5dbbbf75cc-nddkv\" (UID: \"4379e4b2-e68b-4ddf-a3bf-9ec404e61f56\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-nddkv" Nov 02 01:19:58.972952 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:58.972925 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-nddkv" Nov 02 01:19:59.251469 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:59.251249 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-nddkv"] Nov 02 01:19:59.254002 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:19:59.253968 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4379e4b2_e68b_4ddf_a3bf_9ec404e61f56.slice/crio-7da3cca8c5c45593b90c05e099c37b2a041e892ec84d21e986431c419d63a1fa WatchSource:0}: Error finding container 7da3cca8c5c45593b90c05e099c37b2a041e892ec84d21e986431c419d63a1fa: Status 404 returned error can't find the container with id 7da3cca8c5c45593b90c05e099c37b2a041e892ec84d21e986431c419d63a1fa Nov 02 01:19:59.701449 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:19:59.701411 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-nddkv" event={"ID":"4379e4b2-e68b-4ddf-a3bf-9ec404e61f56","Type":"ContainerStarted","Data":"7da3cca8c5c45593b90c05e099c37b2a041e892ec84d21e986431c419d63a1fa"} Nov 02 01:20:01.581393 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:20:01.581340 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:20:04.723607 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:04.723565 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-nddkv" event={"ID":"4379e4b2-e68b-4ddf-a3bf-9ec404e61f56","Type":"ContainerStarted","Data":"a62b71f50d31598e1a3a79e4af7f4ae065b787aa0cf1266cf6b3d5294e7195da"} Nov 02 01:20:04.723990 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:04.723695 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-nddkv" Nov 02 01:20:06.768238 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:06.768132 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-nddkv" podStartSLOduration=3.535409079 podStartE2EDuration="8.768115172s" totalImagesPullingTime="5.232706093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:19:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:20:04.742196673 +0000 UTC m=+309.363201888" watchObservedRunningTime="2026-11-02 01:20:06.768115172 +0000 UTC m=+311.389120383" Nov 02 01:20:06.768648 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:06.768511 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-p9hr8"] Nov 02 01:20:06.771564 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:06.771548 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-p9hr8" Nov 02 01:20:06.782135 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:06.782106 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-zgncr\"" Nov 02 01:20:06.783094 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:06.783076 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-p9hr8"] Nov 02 01:20:06.797205 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:06.797170 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2xk2\" (UniqueName: \"kubernetes.io/projected/e0522085-c7a8-4418-8cc7-d33fd70f957c-kube-api-access-b2xk2\") pod \"cert-manager-cainjector-5c57bbd4fc-p9hr8\" (UID: \"e0522085-c7a8-4418-8cc7-d33fd70f957c\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-p9hr8" Nov 02 01:20:06.797362 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:06.797253 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e0522085-c7a8-4418-8cc7-d33fd70f957c-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-p9hr8\" (UID: \"e0522085-c7a8-4418-8cc7-d33fd70f957c\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-p9hr8" Nov 02 01:20:06.898295 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:06.898251 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2xk2\" (UniqueName: \"kubernetes.io/projected/e0522085-c7a8-4418-8cc7-d33fd70f957c-kube-api-access-b2xk2\") pod \"cert-manager-cainjector-5c57bbd4fc-p9hr8\" (UID: \"e0522085-c7a8-4418-8cc7-d33fd70f957c\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-p9hr8" Nov 02 01:20:06.898506 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:06.898310 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e0522085-c7a8-4418-8cc7-d33fd70f957c-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-p9hr8\" (UID: \"e0522085-c7a8-4418-8cc7-d33fd70f957c\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-p9hr8" Nov 02 01:20:06.930814 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:06.930780 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2xk2\" (UniqueName: \"kubernetes.io/projected/e0522085-c7a8-4418-8cc7-d33fd70f957c-kube-api-access-b2xk2\") pod \"cert-manager-cainjector-5c57bbd4fc-p9hr8\" (UID: \"e0522085-c7a8-4418-8cc7-d33fd70f957c\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-p9hr8" Nov 02 01:20:06.931978 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:06.931954 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e0522085-c7a8-4418-8cc7-d33fd70f957c-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-p9hr8\" (UID: \"e0522085-c7a8-4418-8cc7-d33fd70f957c\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-p9hr8" Nov 02 01:20:07.083250 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:07.083166 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-p9hr8" Nov 02 01:20:07.356500 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:07.356448 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-p9hr8"] Nov 02 01:20:07.359255 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:20:07.359226 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode0522085_c7a8_4418_8cc7_d33fd70f957c.slice/crio-1b6992aa3ab97f7fafb09ffad96467a974b0a8e3c4e72c5535023fc2a9231db9 WatchSource:0}: Error finding container 1b6992aa3ab97f7fafb09ffad96467a974b0a8e3c4e72c5535023fc2a9231db9: Status 404 returned error can't find the container with id 1b6992aa3ab97f7fafb09ffad96467a974b0a8e3c4e72c5535023fc2a9231db9 Nov 02 01:20:07.737070 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:07.737029 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-p9hr8" event={"ID":"e0522085-c7a8-4418-8cc7-d33fd70f957c","Type":"ContainerStarted","Data":"333da2ebcab7cc11e9eb86380deb411660fc96b635da7f2b92f62a661d299733"} Nov 02 01:20:07.737270 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:07.737083 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-p9hr8" event={"ID":"e0522085-c7a8-4418-8cc7-d33fd70f957c","Type":"ContainerStarted","Data":"1b6992aa3ab97f7fafb09ffad96467a974b0a8e3c4e72c5535023fc2a9231db9"} Nov 02 01:20:07.757233 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:07.757179 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-p9hr8" podStartSLOduration=1.757165171 podStartE2EDuration="1.757165171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:20:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:20:07.756040031 +0000 UTC m=+312.377045443" watchObservedRunningTime="2026-11-02 01:20:07.757165171 +0000 UTC m=+312.378170382" Nov 02 01:20:09.894140 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:09.894108 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-rswlp"] Nov 02 01:20:09.897549 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:09.897532 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-rswlp" Nov 02 01:20:09.908054 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:09.908029 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-pzdzz\"" Nov 02 01:20:09.908502 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:09.908483 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-rswlp"] Nov 02 01:20:09.921619 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:09.921592 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qrss\" (UniqueName: \"kubernetes.io/projected/9b38290a-96ca-4005-9c45-5ea875b76934-kube-api-access-7qrss\") pod \"cert-manager-597b95cb84-rswlp\" (UID: \"9b38290a-96ca-4005-9c45-5ea875b76934\") " pod="cert-manager/cert-manager-597b95cb84-rswlp" Nov 02 01:20:09.921733 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:09.921643 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9b38290a-96ca-4005-9c45-5ea875b76934-bound-sa-token\") pod \"cert-manager-597b95cb84-rswlp\" (UID: \"9b38290a-96ca-4005-9c45-5ea875b76934\") " pod="cert-manager/cert-manager-597b95cb84-rswlp" Nov 02 01:20:10.022252 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:10.022219 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9b38290a-96ca-4005-9c45-5ea875b76934-bound-sa-token\") pod \"cert-manager-597b95cb84-rswlp\" (UID: \"9b38290a-96ca-4005-9c45-5ea875b76934\") " pod="cert-manager/cert-manager-597b95cb84-rswlp" Nov 02 01:20:10.022415 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:10.022294 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qrss\" (UniqueName: \"kubernetes.io/projected/9b38290a-96ca-4005-9c45-5ea875b76934-kube-api-access-7qrss\") pod \"cert-manager-597b95cb84-rswlp\" (UID: \"9b38290a-96ca-4005-9c45-5ea875b76934\") " pod="cert-manager/cert-manager-597b95cb84-rswlp" Nov 02 01:20:10.034774 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:10.034744 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9b38290a-96ca-4005-9c45-5ea875b76934-bound-sa-token\") pod \"cert-manager-597b95cb84-rswlp\" (UID: \"9b38290a-96ca-4005-9c45-5ea875b76934\") " pod="cert-manager/cert-manager-597b95cb84-rswlp" Nov 02 01:20:10.040722 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:10.040697 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qrss\" (UniqueName: \"kubernetes.io/projected/9b38290a-96ca-4005-9c45-5ea875b76934-kube-api-access-7qrss\") pod \"cert-manager-597b95cb84-rswlp\" (UID: \"9b38290a-96ca-4005-9c45-5ea875b76934\") " pod="cert-manager/cert-manager-597b95cb84-rswlp" Nov 02 01:20:10.209608 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:10.209522 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-rswlp" Nov 02 01:20:10.482445 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:10.482229 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-rswlp"] Nov 02 01:20:10.485133 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:20:10.485103 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9b38290a_96ca_4005_9c45_5ea875b76934.slice/crio-6aff8fb230847e7ce37db99a2034e9140eb232038ec1dbe6d75c87e8945238a5 WatchSource:0}: Error finding container 6aff8fb230847e7ce37db99a2034e9140eb232038ec1dbe6d75c87e8945238a5: Status 404 returned error can't find the container with id 6aff8fb230847e7ce37db99a2034e9140eb232038ec1dbe6d75c87e8945238a5 Nov 02 01:20:10.729485 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:10.729450 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-nddkv" Nov 02 01:20:10.751660 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:10.751564 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-rswlp" event={"ID":"9b38290a-96ca-4005-9c45-5ea875b76934","Type":"ContainerStarted","Data":"8ae6135f235f4db818130c5028000dcdb983a9153c1baa88c08460374e6d7a1a"} Nov 02 01:20:10.751660 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:10.751612 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-rswlp" event={"ID":"9b38290a-96ca-4005-9c45-5ea875b76934","Type":"ContainerStarted","Data":"6aff8fb230847e7ce37db99a2034e9140eb232038ec1dbe6d75c87e8945238a5"} Nov 02 01:20:10.771509 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:10.771461 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-rswlp" podStartSLOduration=1.771447706 podStartE2EDuration="1.771447706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:20:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:20:10.770978456 +0000 UTC m=+315.391983668" watchObservedRunningTime="2026-11-02 01:20:10.771447706 +0000 UTC m=+315.392452916" Nov 02 01:20:16.313202 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:16.313172 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6c85779f4b-8nj6c"] Nov 02 01:20:16.320873 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:16.320852 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6c85779f4b-8nj6c" Nov 02 01:20:16.328232 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:16.328191 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6c85779f4b-8nj6c"] Nov 02 01:20:16.332330 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:16.332308 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 02 01:20:16.333559 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:16.333537 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 02 01:20:16.375989 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:16.375948 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b74bl\" (UniqueName: \"kubernetes.io/projected/b5f3fd9e-dc3c-4e95-b44d-28565e7a270c-kube-api-access-b74bl\") pod \"kube-auth-proxy-6c85779f4b-8nj6c\" (UID: \"b5f3fd9e-dc3c-4e95-b44d-28565e7a270c\") " pod="openshift-ingress/kube-auth-proxy-6c85779f4b-8nj6c" Nov 02 01:20:16.375989 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:16.375989 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5f3fd9e-dc3c-4e95-b44d-28565e7a270c-tmp\") pod \"kube-auth-proxy-6c85779f4b-8nj6c\" (UID: \"b5f3fd9e-dc3c-4e95-b44d-28565e7a270c\") " pod="openshift-ingress/kube-auth-proxy-6c85779f4b-8nj6c" Nov 02 01:20:16.376236 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:16.376085 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5f3fd9e-dc3c-4e95-b44d-28565e7a270c-tls-certs\") pod \"kube-auth-proxy-6c85779f4b-8nj6c\" (UID: \"b5f3fd9e-dc3c-4e95-b44d-28565e7a270c\") " pod="openshift-ingress/kube-auth-proxy-6c85779f4b-8nj6c" Nov 02 01:20:16.378931 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:16.378906 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 02 01:20:16.379096 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:16.379020 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-6bnmt\"" Nov 02 01:20:16.379147 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:16.379138 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 02 01:20:16.476759 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:16.476723 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b74bl\" (UniqueName: \"kubernetes.io/projected/b5f3fd9e-dc3c-4e95-b44d-28565e7a270c-kube-api-access-b74bl\") pod \"kube-auth-proxy-6c85779f4b-8nj6c\" (UID: \"b5f3fd9e-dc3c-4e95-b44d-28565e7a270c\") " pod="openshift-ingress/kube-auth-proxy-6c85779f4b-8nj6c" Nov 02 01:20:16.476759 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:16.476760 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5f3fd9e-dc3c-4e95-b44d-28565e7a270c-tmp\") pod \"kube-auth-proxy-6c85779f4b-8nj6c\" (UID: \"b5f3fd9e-dc3c-4e95-b44d-28565e7a270c\") " pod="openshift-ingress/kube-auth-proxy-6c85779f4b-8nj6c" Nov 02 01:20:16.476984 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:16.476801 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5f3fd9e-dc3c-4e95-b44d-28565e7a270c-tls-certs\") pod \"kube-auth-proxy-6c85779f4b-8nj6c\" (UID: \"b5f3fd9e-dc3c-4e95-b44d-28565e7a270c\") " pod="openshift-ingress/kube-auth-proxy-6c85779f4b-8nj6c" Nov 02 01:20:16.479289 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:16.479261 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5f3fd9e-dc3c-4e95-b44d-28565e7a270c-tmp\") pod \"kube-auth-proxy-6c85779f4b-8nj6c\" (UID: \"b5f3fd9e-dc3c-4e95-b44d-28565e7a270c\") " pod="openshift-ingress/kube-auth-proxy-6c85779f4b-8nj6c" Nov 02 01:20:16.479484 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:16.479467 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5f3fd9e-dc3c-4e95-b44d-28565e7a270c-tls-certs\") pod \"kube-auth-proxy-6c85779f4b-8nj6c\" (UID: \"b5f3fd9e-dc3c-4e95-b44d-28565e7a270c\") " pod="openshift-ingress/kube-auth-proxy-6c85779f4b-8nj6c" Nov 02 01:20:16.489650 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:16.489627 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b74bl\" (UniqueName: \"kubernetes.io/projected/b5f3fd9e-dc3c-4e95-b44d-28565e7a270c-kube-api-access-b74bl\") pod \"kube-auth-proxy-6c85779f4b-8nj6c\" (UID: \"b5f3fd9e-dc3c-4e95-b44d-28565e7a270c\") " pod="openshift-ingress/kube-auth-proxy-6c85779f4b-8nj6c" Nov 02 01:20:16.638000 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:16.637964 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6c85779f4b-8nj6c" Nov 02 01:20:16.912459 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:16.912437 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6c85779f4b-8nj6c"] Nov 02 01:20:16.915007 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:20:16.914976 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5f3fd9e_dc3c_4e95_b44d_28565e7a270c.slice/crio-1a849dcc34d20850214ffbf87a5483a6f89290308e1d253f8e4fa5c9c5e8f8a6 WatchSource:0}: Error finding container 1a849dcc34d20850214ffbf87a5483a6f89290308e1d253f8e4fa5c9c5e8f8a6: Status 404 returned error can't find the container with id 1a849dcc34d20850214ffbf87a5483a6f89290308e1d253f8e4fa5c9c5e8f8a6 Nov 02 01:20:16.916990 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:16.916967 4427 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 01:20:17.780696 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:17.780656 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6c85779f4b-8nj6c" event={"ID":"b5f3fd9e-dc3c-4e95-b44d-28565e7a270c","Type":"ContainerStarted","Data":"1a849dcc34d20850214ffbf87a5483a6f89290308e1d253f8e4fa5c9c5e8f8a6"} Nov 02 01:20:20.795505 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:20.795457 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6c85779f4b-8nj6c" event={"ID":"b5f3fd9e-dc3c-4e95-b44d-28565e7a270c","Type":"ContainerStarted","Data":"5a972b8b2a73174f631769722f463d60be1baa90e1c43ffc0420eed2df40a4e5"} Nov 02 01:20:20.822545 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:20.822490 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6c85779f4b-8nj6c" podStartSLOduration=1.670599374 podStartE2EDuration="4.822476673s" totalImagesPullingTime="3.151877299s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:20:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:20:20.821297856 +0000 UTC m=+325.442303067" watchObservedRunningTime="2026-11-02 01:20:20.822476673 +0000 UTC m=+325.443481884" Nov 02 01:20:21.976863 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:21.976829 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-ktvrs"] Nov 02 01:20:21.980324 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:21.980307 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-ktvrs" Nov 02 01:20:21.984686 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:21.984659 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 02 01:20:21.984805 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:21.984708 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 02 01:20:21.986101 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:21.986082 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-w92dz\"" Nov 02 01:20:21.986101 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:21.986091 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 02 01:20:21.990609 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:21.990590 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-ktvrs"] Nov 02 01:20:22.022752 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:22.022717 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/df88fb54-9f4e-4725-9634-5a2454c497f2-config-file\") pod \"limitador-limitador-74dfcfb894-ktvrs\" (UID: \"df88fb54-9f4e-4725-9634-5a2454c497f2\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-ktvrs" Nov 02 01:20:22.022904 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:22.022770 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx8rd\" (UniqueName: \"kubernetes.io/projected/df88fb54-9f4e-4725-9634-5a2454c497f2-kube-api-access-mx8rd\") pod \"limitador-limitador-74dfcfb894-ktvrs\" (UID: \"df88fb54-9f4e-4725-9634-5a2454c497f2\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-ktvrs" Nov 02 01:20:22.084528 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:22.084492 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-ktvrs"] Nov 02 01:20:22.123596 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:22.123560 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/df88fb54-9f4e-4725-9634-5a2454c497f2-config-file\") pod \"limitador-limitador-74dfcfb894-ktvrs\" (UID: \"df88fb54-9f4e-4725-9634-5a2454c497f2\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-ktvrs" Nov 02 01:20:22.123749 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:22.123617 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx8rd\" (UniqueName: \"kubernetes.io/projected/df88fb54-9f4e-4725-9634-5a2454c497f2-kube-api-access-mx8rd\") pod \"limitador-limitador-74dfcfb894-ktvrs\" (UID: \"df88fb54-9f4e-4725-9634-5a2454c497f2\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-ktvrs" Nov 02 01:20:22.124229 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:22.124208 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/df88fb54-9f4e-4725-9634-5a2454c497f2-config-file\") pod \"limitador-limitador-74dfcfb894-ktvrs\" (UID: \"df88fb54-9f4e-4725-9634-5a2454c497f2\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-ktvrs" Nov 02 01:20:22.137904 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:22.137869 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx8rd\" (UniqueName: \"kubernetes.io/projected/df88fb54-9f4e-4725-9634-5a2454c497f2-kube-api-access-mx8rd\") pod \"limitador-limitador-74dfcfb894-ktvrs\" (UID: \"df88fb54-9f4e-4725-9634-5a2454c497f2\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-ktvrs" Nov 02 01:20:22.293063 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:22.292975 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-ktvrs" Nov 02 01:20:22.493192 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:22.493162 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-nr4n2"] Nov 02 01:20:22.496623 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:22.496607 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-nr4n2" Nov 02 01:20:22.504246 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:22.503395 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Nov 02 01:20:22.505019 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:22.504528 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Nov 02 01:20:22.505598 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:22.505549 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-nr4n2"] Nov 02 01:20:22.526924 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:22.526898 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh2fk\" (UniqueName: \"kubernetes.io/projected/65197f55-7074-436e-a8b1-320105fde335-kube-api-access-bh2fk\") pod \"kuadrant-console-plugin-66c4fb575d-nr4n2\" (UID: \"65197f55-7074-436e-a8b1-320105fde335\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-nr4n2" Nov 02 01:20:22.527068 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:22.526959 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/65197f55-7074-436e-a8b1-320105fde335-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-nr4n2\" (UID: \"65197f55-7074-436e-a8b1-320105fde335\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-nr4n2" Nov 02 01:20:22.527068 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:22.527027 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/65197f55-7074-436e-a8b1-320105fde335-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-nr4n2\" (UID: \"65197f55-7074-436e-a8b1-320105fde335\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-nr4n2" Nov 02 01:20:22.565317 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:22.565264 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-ktvrs"] Nov 02 01:20:22.567573 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:20:22.567544 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf88fb54_9f4e_4725_9634_5a2454c497f2.slice/crio-1eeb9dc156306c85ff3d0ebeecee9be0fccbdabd9be7ae76659ca516a954e0ac WatchSource:0}: Error finding container 1eeb9dc156306c85ff3d0ebeecee9be0fccbdabd9be7ae76659ca516a954e0ac: Status 404 returned error can't find the container with id 1eeb9dc156306c85ff3d0ebeecee9be0fccbdabd9be7ae76659ca516a954e0ac Nov 02 01:20:22.627944 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:22.627907 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh2fk\" (UniqueName: \"kubernetes.io/projected/65197f55-7074-436e-a8b1-320105fde335-kube-api-access-bh2fk\") pod \"kuadrant-console-plugin-66c4fb575d-nr4n2\" (UID: \"65197f55-7074-436e-a8b1-320105fde335\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-nr4n2" Nov 02 01:20:22.628133 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:22.627970 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/65197f55-7074-436e-a8b1-320105fde335-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-nr4n2\" (UID: \"65197f55-7074-436e-a8b1-320105fde335\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-nr4n2" Nov 02 01:20:22.628133 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:22.627996 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/65197f55-7074-436e-a8b1-320105fde335-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-nr4n2\" (UID: \"65197f55-7074-436e-a8b1-320105fde335\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-nr4n2" Nov 02 01:20:22.628799 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:22.628731 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/65197f55-7074-436e-a8b1-320105fde335-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-nr4n2\" (UID: \"65197f55-7074-436e-a8b1-320105fde335\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-nr4n2" Nov 02 01:20:22.630570 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:22.630551 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/65197f55-7074-436e-a8b1-320105fde335-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-nr4n2\" (UID: \"65197f55-7074-436e-a8b1-320105fde335\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-nr4n2" Nov 02 01:20:22.640209 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:22.640189 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh2fk\" (UniqueName: \"kubernetes.io/projected/65197f55-7074-436e-a8b1-320105fde335-kube-api-access-bh2fk\") pod \"kuadrant-console-plugin-66c4fb575d-nr4n2\" (UID: \"65197f55-7074-436e-a8b1-320105fde335\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-nr4n2" Nov 02 01:20:22.804384 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:22.804345 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-ktvrs" event={"ID":"df88fb54-9f4e-4725-9634-5a2454c497f2","Type":"ContainerStarted","Data":"1eeb9dc156306c85ff3d0ebeecee9be0fccbdabd9be7ae76659ca516a954e0ac"} Nov 02 01:20:22.810425 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:22.810399 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-nr4n2" Nov 02 01:20:22.979313 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:22.979277 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-hhbp4"] Nov 02 01:20:22.982714 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:22.982696 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-hhbp4" Nov 02 01:20:22.990635 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:22.990613 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-6lfnj\"" Nov 02 01:20:22.992046 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:22.992020 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-hhbp4"] Nov 02 01:20:23.032000 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:23.031967 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tp8r\" (UniqueName: \"kubernetes.io/projected/5204ad71-2e13-4225-90fd-9ee6c34831cf-kube-api-access-8tp8r\") pod \"authorino-56c6c9ddd5-hhbp4\" (UID: \"5204ad71-2e13-4225-90fd-9ee6c34831cf\") " pod="kuadrant-system/authorino-56c6c9ddd5-hhbp4" Nov 02 01:20:23.082753 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:23.081740 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-nr4n2"] Nov 02 01:20:23.133060 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:23.133027 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tp8r\" (UniqueName: \"kubernetes.io/projected/5204ad71-2e13-4225-90fd-9ee6c34831cf-kube-api-access-8tp8r\") pod \"authorino-56c6c9ddd5-hhbp4\" (UID: \"5204ad71-2e13-4225-90fd-9ee6c34831cf\") " pod="kuadrant-system/authorino-56c6c9ddd5-hhbp4" Nov 02 01:20:23.145328 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:23.145300 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tp8r\" (UniqueName: \"kubernetes.io/projected/5204ad71-2e13-4225-90fd-9ee6c34831cf-kube-api-access-8tp8r\") pod \"authorino-56c6c9ddd5-hhbp4\" (UID: \"5204ad71-2e13-4225-90fd-9ee6c34831cf\") " pod="kuadrant-system/authorino-56c6c9ddd5-hhbp4" Nov 02 01:20:23.296216 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:23.296179 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-hhbp4" Nov 02 01:20:23.567538 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:23.567502 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-hhbp4"] Nov 02 01:20:23.569708 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:20:23.569675 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5204ad71_2e13_4225_90fd_9ee6c34831cf.slice/crio-ec4d73fd82f8c6d2f0a384398be948d0a90a199a495ef864c4ff65eb9bdffb5c WatchSource:0}: Error finding container ec4d73fd82f8c6d2f0a384398be948d0a90a199a495ef864c4ff65eb9bdffb5c: Status 404 returned error can't find the container with id ec4d73fd82f8c6d2f0a384398be948d0a90a199a495ef864c4ff65eb9bdffb5c Nov 02 01:20:23.810230 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:23.810191 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-hhbp4" event={"ID":"5204ad71-2e13-4225-90fd-9ee6c34831cf","Type":"ContainerStarted","Data":"ec4d73fd82f8c6d2f0a384398be948d0a90a199a495ef864c4ff65eb9bdffb5c"} Nov 02 01:20:23.811340 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:23.811316 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-nr4n2" event={"ID":"65197f55-7074-436e-a8b1-320105fde335","Type":"ContainerStarted","Data":"56518023a8070334b458c6b5418389aebf3e5bfcd52738323b4be392041212ea"} Nov 02 01:20:30.877196 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:30.877150 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-hhbp4" event={"ID":"5204ad71-2e13-4225-90fd-9ee6c34831cf","Type":"ContainerStarted","Data":"8ce95b2151863cc80791db9f87c53bc33e4df496aeaa26f41222f5b8b6b5e68d"} Nov 02 01:20:30.878635 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:30.878613 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-nr4n2" event={"ID":"65197f55-7074-436e-a8b1-320105fde335","Type":"ContainerStarted","Data":"52e635cbd1deaf5c13f6df7dc9d69d374156a7e89048cd9659adf68de82e8821"} Nov 02 01:20:30.879980 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:30.879959 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-ktvrs" event={"ID":"df88fb54-9f4e-4725-9634-5a2454c497f2","Type":"ContainerStarted","Data":"3a45077d7ce4072ca345b31c65cda51f69d3edd52cce0387e74170c98767be16"} Nov 02 01:20:30.880041 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:30.880015 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-ktvrs" Nov 02 01:20:30.894315 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:30.894275 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-hhbp4" podStartSLOduration=2.622380483 podStartE2EDuration="8.894262991s" totalImagesPullingTime="6.271882508s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:20:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:20:30.893260039 +0000 UTC m=+335.514265251" watchObservedRunningTime="2026-11-02 01:20:30.894262991 +0000 UTC m=+335.515268202" Nov 02 01:20:30.911846 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:30.911799 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-ktvrs" podStartSLOduration=2.117706263 podStartE2EDuration="9.911784405s" totalImagesPullingTime="7.794078142s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:20:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:20:30.910561132 +0000 UTC m=+335.531566344" watchObservedRunningTime="2026-11-02 01:20:30.911784405 +0000 UTC m=+335.532789618" Nov 02 01:20:30.930278 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:30.930229 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-nr4n2" podStartSLOduration=1.493679541 podStartE2EDuration="8.930214477s" totalImagesPullingTime="7.436534936s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:20:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:20:30.929671191 +0000 UTC m=+335.550676402" watchObservedRunningTime="2026-11-02 01:20:30.930214477 +0000 UTC m=+335.551219687" Nov 02 01:20:31.580957 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:20:31.580919 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:20:36.192037 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:36.192000 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-ktvrs"] Nov 02 01:20:36.192469 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:36.192183 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-ktvrs" podUID="df88fb54-9f4e-4725-9634-5a2454c497f2" containerName="limitador" containerID="cri-o://3a45077d7ce4072ca345b31c65cda51f69d3edd52cce0387e74170c98767be16" gracePeriod=30 Nov 02 01:20:36.194324 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:36.194300 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-ktvrs" Nov 02 01:20:36.774944 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:36.774867 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-ktvrs" Nov 02 01:20:36.854720 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:36.854686 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df88fb54-9f4e-4725-9634-5a2454c497f2-config-file\" (UniqueName: \"kubernetes.io/configmap/df88fb54-9f4e-4725-9634-5a2454c497f2-config-file\") pod \"df88fb54-9f4e-4725-9634-5a2454c497f2\" (UID: \"df88fb54-9f4e-4725-9634-5a2454c497f2\") " Nov 02 01:20:36.854720 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:36.854728 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df88fb54-9f4e-4725-9634-5a2454c497f2-kube-api-access-mx8rd\" (UniqueName: \"kubernetes.io/projected/df88fb54-9f4e-4725-9634-5a2454c497f2-kube-api-access-mx8rd\") pod \"df88fb54-9f4e-4725-9634-5a2454c497f2\" (UID: \"df88fb54-9f4e-4725-9634-5a2454c497f2\") " Nov 02 01:20:36.855081 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:36.855057 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df88fb54-9f4e-4725-9634-5a2454c497f2-config-file" pod "df88fb54-9f4e-4725-9634-5a2454c497f2" (UID: "df88fb54-9f4e-4725-9634-5a2454c497f2"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:20:36.857091 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:36.857068 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df88fb54-9f4e-4725-9634-5a2454c497f2-kube-api-access-mx8rd" pod "df88fb54-9f4e-4725-9634-5a2454c497f2" (UID: "df88fb54-9f4e-4725-9634-5a2454c497f2"). InnerVolumeSpecName "kube-api-access-mx8rd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:20:36.904144 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:36.904110 4427 generic.go:356] "Generic (PLEG): container finished" podID="df88fb54-9f4e-4725-9634-5a2454c497f2" containerID="3a45077d7ce4072ca345b31c65cda51f69d3edd52cce0387e74170c98767be16" exitCode=0 Nov 02 01:20:36.904301 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:36.904161 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-ktvrs" Nov 02 01:20:36.904301 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:36.904185 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-ktvrs" event={"ID":"df88fb54-9f4e-4725-9634-5a2454c497f2","Type":"ContainerDied","Data":"3a45077d7ce4072ca345b31c65cda51f69d3edd52cce0387e74170c98767be16"} Nov 02 01:20:36.904301 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:36.904239 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-ktvrs" event={"ID":"df88fb54-9f4e-4725-9634-5a2454c497f2","Type":"ContainerDied","Data":"1eeb9dc156306c85ff3d0ebeecee9be0fccbdabd9be7ae76659ca516a954e0ac"} Nov 02 01:20:36.904301 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:36.904261 4427 scope.go:122] "RemoveContainer" containerID="3a45077d7ce4072ca345b31c65cda51f69d3edd52cce0387e74170c98767be16" Nov 02 01:20:36.913428 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:36.913410 4427 scope.go:122] "RemoveContainer" containerID="3a45077d7ce4072ca345b31c65cda51f69d3edd52cce0387e74170c98767be16" Nov 02 01:20:36.913713 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:20:36.913688 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a45077d7ce4072ca345b31c65cda51f69d3edd52cce0387e74170c98767be16\": container with ID starting with 3a45077d7ce4072ca345b31c65cda51f69d3edd52cce0387e74170c98767be16 not found: ID does not exist" containerID="3a45077d7ce4072ca345b31c65cda51f69d3edd52cce0387e74170c98767be16" Nov 02 01:20:36.913792 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:36.913725 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a45077d7ce4072ca345b31c65cda51f69d3edd52cce0387e74170c98767be16"} err="failed to get container status \"3a45077d7ce4072ca345b31c65cda51f69d3edd52cce0387e74170c98767be16\": rpc error: code = NotFound desc = could not find container \"3a45077d7ce4072ca345b31c65cda51f69d3edd52cce0387e74170c98767be16\": container with ID starting with 3a45077d7ce4072ca345b31c65cda51f69d3edd52cce0387e74170c98767be16 not found: ID does not exist" Nov 02 01:20:36.925425 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:36.925394 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-ktvrs"] Nov 02 01:20:36.929410 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:36.929388 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-ktvrs"] Nov 02 01:20:36.955606 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:36.955581 4427 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/df88fb54-9f4e-4725-9634-5a2454c497f2-config-file\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:20:36.955606 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:36.955605 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx8rd\" (UniqueName: \"kubernetes.io/projected/df88fb54-9f4e-4725-9634-5a2454c497f2-kube-api-access-mx8rd\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:20:37.531941 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:37.531907 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df88fb54-9f4e-4725-9634-5a2454c497f2" path="/var/lib/kubelet/pods/df88fb54-9f4e-4725-9634-5a2454c497f2/volumes" Nov 02 01:20:39.288549 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:39.288522 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-7z6rk"] Nov 02 01:20:39.288962 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:39.288838 4427 cpu_manager.go:414] "containerMap: removing container" podUID="df88fb54-9f4e-4725-9634-5a2454c497f2" containerName="limitador" Nov 02 01:20:39.288962 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:39.288848 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df88fb54-9f4e-4725-9634-5a2454c497f2" containerName="limitador" Nov 02 01:20:39.288962 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:39.288926 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="df88fb54-9f4e-4725-9634-5a2454c497f2" containerName="limitador" Nov 02 01:20:39.291818 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:39.291797 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-7z6rk" Nov 02 01:20:39.295433 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:39.295414 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Nov 02 01:20:39.299113 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:39.299087 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 01:20:39.299231 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:39.299107 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 01:20:39.300718 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:39.300697 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-7z6rk"] Nov 02 01:20:39.379632 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:39.379585 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4d514a34-b384-47a4-9193-c475aefc8447-data\") pod \"postgres-7cdbf6dc44-7z6rk\" (UID: \"4d514a34-b384-47a4-9193-c475aefc8447\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-7z6rk" Nov 02 01:20:39.379810 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:39.379655 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9pt5\" (UniqueName: \"kubernetes.io/projected/4d514a34-b384-47a4-9193-c475aefc8447-kube-api-access-h9pt5\") pod \"postgres-7cdbf6dc44-7z6rk\" (UID: \"4d514a34-b384-47a4-9193-c475aefc8447\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-7z6rk" Nov 02 01:20:39.395838 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:39.395809 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-h42cg\"" Nov 02 01:20:39.480760 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:39.480725 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4d514a34-b384-47a4-9193-c475aefc8447-data\") pod \"postgres-7cdbf6dc44-7z6rk\" (UID: \"4d514a34-b384-47a4-9193-c475aefc8447\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-7z6rk" Nov 02 01:20:39.480904 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:39.480780 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9pt5\" (UniqueName: \"kubernetes.io/projected/4d514a34-b384-47a4-9193-c475aefc8447-kube-api-access-h9pt5\") pod \"postgres-7cdbf6dc44-7z6rk\" (UID: \"4d514a34-b384-47a4-9193-c475aefc8447\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-7z6rk" Nov 02 01:20:39.481147 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:39.481127 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4d514a34-b384-47a4-9193-c475aefc8447-data\") pod \"postgres-7cdbf6dc44-7z6rk\" (UID: \"4d514a34-b384-47a4-9193-c475aefc8447\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-7z6rk" Nov 02 01:20:39.493126 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:39.493089 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9pt5\" (UniqueName: \"kubernetes.io/projected/4d514a34-b384-47a4-9193-c475aefc8447-kube-api-access-h9pt5\") pod \"postgres-7cdbf6dc44-7z6rk\" (UID: \"4d514a34-b384-47a4-9193-c475aefc8447\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-7z6rk" Nov 02 01:20:39.605600 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:39.605509 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-7z6rk" Nov 02 01:20:39.877886 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:39.877862 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-7z6rk"] Nov 02 01:20:39.879915 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:20:39.879889 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d514a34_b384_47a4_9193_c475aefc8447.slice/crio-2bf8d3fb17284ab496833e0a6005e130f2112fea552086a19e3bcea690104b2e WatchSource:0}: Error finding container 2bf8d3fb17284ab496833e0a6005e130f2112fea552086a19e3bcea690104b2e: Status 404 returned error can't find the container with id 2bf8d3fb17284ab496833e0a6005e130f2112fea552086a19e3bcea690104b2e Nov 02 01:20:39.918820 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:39.918783 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-7z6rk" event={"ID":"4d514a34-b384-47a4-9193-c475aefc8447","Type":"ContainerStarted","Data":"2bf8d3fb17284ab496833e0a6005e130f2112fea552086a19e3bcea690104b2e"} Nov 02 01:20:45.949526 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:45.949482 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-7z6rk" event={"ID":"4d514a34-b384-47a4-9193-c475aefc8447","Type":"ContainerStarted","Data":"5fc814fdd97e49aa507a64e2bc51fbc0c04197b0045808ab781bffab76ebc9c0"} Nov 02 01:20:45.949934 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:45.949587 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-7z6rk" Nov 02 01:20:45.967747 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:45.967695 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-7z6rk" podStartSLOduration=1.085476178 podStartE2EDuration="6.967681261s" totalImagesPullingTime="5.882205083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:20:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:20:45.966627062 +0000 UTC m=+350.587632273" watchObservedRunningTime="2026-11-02 01:20:45.967681261 +0000 UTC m=+350.588686472" Nov 02 01:20:51.983800 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:51.983768 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-7z6rk" Nov 02 01:20:59.251234 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:59.251197 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-587977945b-ktcgw"] Nov 02 01:20:59.417214 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:59.417177 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-587977945b-ktcgw"] Nov 02 01:20:59.417409 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:59.417322 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-587977945b-ktcgw" Nov 02 01:20:59.423351 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:59.423329 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 02 01:20:59.423492 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:59.423330 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 02 01:20:59.423492 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:59.423430 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ldcrm\"" Nov 02 01:20:59.423492 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:59.423347 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 02 01:20:59.423492 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:59.423347 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 02 01:20:59.423492 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:59.423489 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 02 01:20:59.562555 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:59.562464 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bcf15fa4-15f1-4dc9-8755-9bcccfdc846b-webhook-cert\") pod \"maas-controller-587977945b-ktcgw\" (UID: \"bcf15fa4-15f1-4dc9-8755-9bcccfdc846b\") " pod="opendatahub/maas-controller-587977945b-ktcgw" Nov 02 01:20:59.562555 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:59.562517 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcf15fa4-15f1-4dc9-8755-9bcccfdc846b-metrics-certs\") pod \"maas-controller-587977945b-ktcgw\" (UID: \"bcf15fa4-15f1-4dc9-8755-9bcccfdc846b\") " pod="opendatahub/maas-controller-587977945b-ktcgw" Nov 02 01:20:59.562555 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:59.562541 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh8vb\" (UniqueName: \"kubernetes.io/projected/bcf15fa4-15f1-4dc9-8755-9bcccfdc846b-kube-api-access-dh8vb\") pod \"maas-controller-587977945b-ktcgw\" (UID: \"bcf15fa4-15f1-4dc9-8755-9bcccfdc846b\") " pod="opendatahub/maas-controller-587977945b-ktcgw" Nov 02 01:20:59.663828 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:59.663794 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bcf15fa4-15f1-4dc9-8755-9bcccfdc846b-webhook-cert\") pod \"maas-controller-587977945b-ktcgw\" (UID: \"bcf15fa4-15f1-4dc9-8755-9bcccfdc846b\") " pod="opendatahub/maas-controller-587977945b-ktcgw" Nov 02 01:20:59.664026 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:59.663959 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcf15fa4-15f1-4dc9-8755-9bcccfdc846b-metrics-certs\") pod \"maas-controller-587977945b-ktcgw\" (UID: \"bcf15fa4-15f1-4dc9-8755-9bcccfdc846b\") " pod="opendatahub/maas-controller-587977945b-ktcgw" Nov 02 01:20:59.664026 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:59.663994 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh8vb\" (UniqueName: \"kubernetes.io/projected/bcf15fa4-15f1-4dc9-8755-9bcccfdc846b-kube-api-access-dh8vb\") pod \"maas-controller-587977945b-ktcgw\" (UID: \"bcf15fa4-15f1-4dc9-8755-9bcccfdc846b\") " pod="opendatahub/maas-controller-587977945b-ktcgw" Nov 02 01:20:59.666449 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:59.666425 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcf15fa4-15f1-4dc9-8755-9bcccfdc846b-metrics-certs\") pod \"maas-controller-587977945b-ktcgw\" (UID: \"bcf15fa4-15f1-4dc9-8755-9bcccfdc846b\") " pod="opendatahub/maas-controller-587977945b-ktcgw" Nov 02 01:20:59.666449 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:59.666435 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bcf15fa4-15f1-4dc9-8755-9bcccfdc846b-webhook-cert\") pod \"maas-controller-587977945b-ktcgw\" (UID: \"bcf15fa4-15f1-4dc9-8755-9bcccfdc846b\") " pod="opendatahub/maas-controller-587977945b-ktcgw" Nov 02 01:20:59.676728 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:59.676698 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh8vb\" (UniqueName: \"kubernetes.io/projected/bcf15fa4-15f1-4dc9-8755-9bcccfdc846b-kube-api-access-dh8vb\") pod \"maas-controller-587977945b-ktcgw\" (UID: \"bcf15fa4-15f1-4dc9-8755-9bcccfdc846b\") " pod="opendatahub/maas-controller-587977945b-ktcgw" Nov 02 01:20:59.771636 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:20:59.771593 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-587977945b-ktcgw" Nov 02 01:21:00.076144 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:00.076121 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-587977945b-ktcgw"] Nov 02 01:21:00.078467 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:21:00.078428 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcf15fa4_15f1_4dc9_8755_9bcccfdc846b.slice/crio-029c0899ce6e99f9caea3937c30a40a1c947deb51dca3cae3c024ac90533ff7d WatchSource:0}: Error finding container 029c0899ce6e99f9caea3937c30a40a1c947deb51dca3cae3c024ac90533ff7d: Status 404 returned error can't find the container with id 029c0899ce6e99f9caea3937c30a40a1c947deb51dca3cae3c024ac90533ff7d Nov 02 01:21:01.012679 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:01.012631 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-587977945b-ktcgw" event={"ID":"bcf15fa4-15f1-4dc9-8755-9bcccfdc846b","Type":"ContainerStarted","Data":"029c0899ce6e99f9caea3937c30a40a1c947deb51dca3cae3c024ac90533ff7d"} Nov 02 01:21:01.584559 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:21:01.584517 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:21:03.024234 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:03.024148 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-587977945b-ktcgw" event={"ID":"bcf15fa4-15f1-4dc9-8755-9bcccfdc846b","Type":"ContainerStarted","Data":"c22144c1728a9529c65802199385bc79bf0d1822b08839703133f102f0153502"} Nov 02 01:21:03.024660 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:03.024310 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-587977945b-ktcgw" Nov 02 01:21:03.048265 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:03.048210 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-587977945b-ktcgw" podStartSLOduration=1.466970324 podStartE2EDuration="4.048195816s" totalImagesPullingTime="2.581225492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:20:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:21:03.047111462 +0000 UTC m=+367.668116675" watchObservedRunningTime="2026-11-02 01:21:03.048195816 +0000 UTC m=+367.669201044" Nov 02 01:21:14.030900 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:14.030863 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-587977945b-ktcgw" Nov 02 01:21:14.674480 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:14.674442 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-xh22d"] Nov 02 01:21:14.677903 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:14.677881 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xh22d" Nov 02 01:21:14.690173 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:14.690080 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-xh22d"] Nov 02 01:21:14.690731 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:14.690708 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Nov 02 01:21:14.690839 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:14.690794 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-8d6lg\"" Nov 02 01:21:14.692302 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:14.692279 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b8c9ab8f-5d15-497f-9365-b72bdbb46cc9-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-xh22d\" (UID: \"b8c9ab8f-5d15-497f-9365-b72bdbb46cc9\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xh22d" Nov 02 01:21:14.692418 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:14.692330 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2txq\" (UniqueName: \"kubernetes.io/projected/b8c9ab8f-5d15-497f-9365-b72bdbb46cc9-kube-api-access-b2txq\") pod \"llmisvc-controller-manager-869d5dcc5d-xh22d\" (UID: \"b8c9ab8f-5d15-497f-9365-b72bdbb46cc9\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xh22d" Nov 02 01:21:14.719833 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:14.719801 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-45dsr"] Nov 02 01:21:14.723222 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:14.723203 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-45dsr" Nov 02 01:21:14.735862 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:14.735839 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-45dsr"] Nov 02 01:21:14.752953 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:14.752902 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Nov 02 01:21:14.753171 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:14.753112 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Nov 02 01:21:14.753269 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:14.753181 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-zhszv\"" Nov 02 01:21:14.793706 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:14.793677 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2txq\" (UniqueName: \"kubernetes.io/projected/b8c9ab8f-5d15-497f-9365-b72bdbb46cc9-kube-api-access-b2txq\") pod \"llmisvc-controller-manager-869d5dcc5d-xh22d\" (UID: \"b8c9ab8f-5d15-497f-9365-b72bdbb46cc9\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xh22d" Nov 02 01:21:14.793902 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:14.793716 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwn57\" (UniqueName: \"kubernetes.io/projected/e8c52127-2f29-46e8-8c99-ce29cfc89614-kube-api-access-dwn57\") pod \"odh-model-controller-769875db56-45dsr\" (UID: \"e8c52127-2f29-46e8-8c99-ce29cfc89614\") " pod="opendatahub/odh-model-controller-769875db56-45dsr" Nov 02 01:21:14.793902 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:14.793788 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e8c52127-2f29-46e8-8c99-ce29cfc89614-cert\") pod \"odh-model-controller-769875db56-45dsr\" (UID: \"e8c52127-2f29-46e8-8c99-ce29cfc89614\") " pod="opendatahub/odh-model-controller-769875db56-45dsr" Nov 02 01:21:14.793902 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:14.793817 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b8c9ab8f-5d15-497f-9365-b72bdbb46cc9-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-xh22d\" (UID: \"b8c9ab8f-5d15-497f-9365-b72bdbb46cc9\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xh22d" Nov 02 01:21:14.793902 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:14.793836 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8c52127-2f29-46e8-8c99-ce29cfc89614-metrics-certs\") pod \"odh-model-controller-769875db56-45dsr\" (UID: \"e8c52127-2f29-46e8-8c99-ce29cfc89614\") " pod="opendatahub/odh-model-controller-769875db56-45dsr" Nov 02 01:21:14.796655 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:14.796624 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b8c9ab8f-5d15-497f-9365-b72bdbb46cc9-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-xh22d\" (UID: \"b8c9ab8f-5d15-497f-9365-b72bdbb46cc9\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xh22d" Nov 02 01:21:14.806470 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:14.806449 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2txq\" (UniqueName: \"kubernetes.io/projected/b8c9ab8f-5d15-497f-9365-b72bdbb46cc9-kube-api-access-b2txq\") pod \"llmisvc-controller-manager-869d5dcc5d-xh22d\" (UID: \"b8c9ab8f-5d15-497f-9365-b72bdbb46cc9\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xh22d" Nov 02 01:21:14.894678 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:14.894643 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e8c52127-2f29-46e8-8c99-ce29cfc89614-cert\") pod \"odh-model-controller-769875db56-45dsr\" (UID: \"e8c52127-2f29-46e8-8c99-ce29cfc89614\") " pod="opendatahub/odh-model-controller-769875db56-45dsr" Nov 02 01:21:14.894854 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:14.894689 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8c52127-2f29-46e8-8c99-ce29cfc89614-metrics-certs\") pod \"odh-model-controller-769875db56-45dsr\" (UID: \"e8c52127-2f29-46e8-8c99-ce29cfc89614\") " pod="opendatahub/odh-model-controller-769875db56-45dsr" Nov 02 01:21:14.894854 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:14.894718 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwn57\" (UniqueName: \"kubernetes.io/projected/e8c52127-2f29-46e8-8c99-ce29cfc89614-kube-api-access-dwn57\") pod \"odh-model-controller-769875db56-45dsr\" (UID: \"e8c52127-2f29-46e8-8c99-ce29cfc89614\") " pod="opendatahub/odh-model-controller-769875db56-45dsr" Nov 02 01:21:14.897228 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:14.897200 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e8c52127-2f29-46e8-8c99-ce29cfc89614-cert\") pod \"odh-model-controller-769875db56-45dsr\" (UID: \"e8c52127-2f29-46e8-8c99-ce29cfc89614\") " pod="opendatahub/odh-model-controller-769875db56-45dsr" Nov 02 01:21:14.897338 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:14.897321 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8c52127-2f29-46e8-8c99-ce29cfc89614-metrics-certs\") pod \"odh-model-controller-769875db56-45dsr\" (UID: \"e8c52127-2f29-46e8-8c99-ce29cfc89614\") " pod="opendatahub/odh-model-controller-769875db56-45dsr" Nov 02 01:21:14.910198 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:14.910170 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwn57\" (UniqueName: \"kubernetes.io/projected/e8c52127-2f29-46e8-8c99-ce29cfc89614-kube-api-access-dwn57\") pod \"odh-model-controller-769875db56-45dsr\" (UID: \"e8c52127-2f29-46e8-8c99-ce29cfc89614\") " pod="opendatahub/odh-model-controller-769875db56-45dsr" Nov 02 01:21:14.990326 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:14.990239 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xh22d" Nov 02 01:21:15.035571 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:15.035534 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-45dsr" Nov 02 01:21:15.314857 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:15.314773 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-xh22d"] Nov 02 01:21:15.318038 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:21:15.318006 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb8c9ab8f_5d15_497f_9365_b72bdbb46cc9.slice/crio-af71bfcc05553dc8b966e4594cdb64cb2cd1435609d73d3d0fe50b755702e801 WatchSource:0}: Error finding container af71bfcc05553dc8b966e4594cdb64cb2cd1435609d73d3d0fe50b755702e801: Status 404 returned error can't find the container with id af71bfcc05553dc8b966e4594cdb64cb2cd1435609d73d3d0fe50b755702e801 Nov 02 01:21:15.349165 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:15.349129 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-45dsr"] Nov 02 01:21:15.352154 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:21:15.352125 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8c52127_2f29_46e8_8c99_ce29cfc89614.slice/crio-ba5626be0cfdcbd9fb92ccbd3de5bdbb2a96a913b6a5d65258fcb68685285a26 WatchSource:0}: Error finding container ba5626be0cfdcbd9fb92ccbd3de5bdbb2a96a913b6a5d65258fcb68685285a26: Status 404 returned error can't find the container with id ba5626be0cfdcbd9fb92ccbd3de5bdbb2a96a913b6a5d65258fcb68685285a26 Nov 02 01:21:16.083724 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:16.083683 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-45dsr" event={"ID":"e8c52127-2f29-46e8-8c99-ce29cfc89614","Type":"ContainerStarted","Data":"ba5626be0cfdcbd9fb92ccbd3de5bdbb2a96a913b6a5d65258fcb68685285a26"} Nov 02 01:21:16.084902 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:16.084873 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xh22d" event={"ID":"b8c9ab8f-5d15-497f-9365-b72bdbb46cc9","Type":"ContainerStarted","Data":"af71bfcc05553dc8b966e4594cdb64cb2cd1435609d73d3d0fe50b755702e801"} Nov 02 01:21:19.103293 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:19.103236 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-45dsr" event={"ID":"e8c52127-2f29-46e8-8c99-ce29cfc89614","Type":"ContainerStarted","Data":"22471d5ff3966ba2ea92991cf5c2a39f24e9fa8f45bead4b588925999bab718a"} Nov 02 01:21:19.103736 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:19.103492 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-45dsr" Nov 02 01:21:19.124856 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:19.124793 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-45dsr" podStartSLOduration=1.5478793899999999 podStartE2EDuration="5.124774949s" totalImagesPullingTime="3.576895559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:21:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:21:19.122179816 +0000 UTC m=+383.743185028" watchObservedRunningTime="2026-11-02 01:21:19.124774949 +0000 UTC m=+383.745780163" Nov 02 01:21:20.112417 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:20.112353 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xh22d" event={"ID":"b8c9ab8f-5d15-497f-9365-b72bdbb46cc9","Type":"ContainerStarted","Data":"515ebc453f8f9d1772dfd5b2b0287b9daef0a0fdab332d6c4a75766b6f5e1088"} Nov 02 01:21:20.112882 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:20.112698 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xh22d" Nov 02 01:21:20.132089 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:20.132012 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xh22d" podStartSLOduration=1.773896457 podStartE2EDuration="6.131997356s" totalImagesPullingTime="4.358100899s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:21:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:21:20.13055238 +0000 UTC m=+384.751557590" watchObservedRunningTime="2026-11-02 01:21:20.131997356 +0000 UTC m=+384.753002568" Nov 02 01:21:30.114571 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:30.114537 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-45dsr" Nov 02 01:21:31.430761 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:31.430726 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-plj5l"] Nov 02 01:21:31.434649 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:31.434626 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-plj5l" Nov 02 01:21:31.445730 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:31.445438 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 01:21:31.445730 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:31.445592 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 02 01:21:31.447521 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:31.447499 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-plj5l"] Nov 02 01:21:31.549572 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:31.549534 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ac7faef-5a37-4ac9-9513-958f0e4650ff-openshift-service-ca\") pod \"maas-api-6d669c7fcd-plj5l\" (UID: \"0ac7faef-5a37-4ac9-9513-958f0e4650ff\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-plj5l" Nov 02 01:21:31.549735 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:31.549587 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnrmf\" (UniqueName: \"kubernetes.io/projected/0ac7faef-5a37-4ac9-9513-958f0e4650ff-kube-api-access-dnrmf\") pod \"maas-api-6d669c7fcd-plj5l\" (UID: \"0ac7faef-5a37-4ac9-9513-958f0e4650ff\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-plj5l" Nov 02 01:21:31.549735 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:31.549699 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ac7faef-5a37-4ac9-9513-958f0e4650ff-maas-api-tls\") pod \"maas-api-6d669c7fcd-plj5l\" (UID: \"0ac7faef-5a37-4ac9-9513-958f0e4650ff\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-plj5l" Nov 02 01:21:31.549822 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:31.549755 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ac7faef-5a37-4ac9-9513-958f0e4650ff-metrics-certs\") pod \"maas-api-6d669c7fcd-plj5l\" (UID: \"0ac7faef-5a37-4ac9-9513-958f0e4650ff\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-plj5l" Nov 02 01:21:31.581052 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:21:31.581012 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:21:31.651250 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:31.651209 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ac7faef-5a37-4ac9-9513-958f0e4650ff-metrics-certs\") pod \"maas-api-6d669c7fcd-plj5l\" (UID: \"0ac7faef-5a37-4ac9-9513-958f0e4650ff\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-plj5l" Nov 02 01:21:31.651492 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:31.651286 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ac7faef-5a37-4ac9-9513-958f0e4650ff-openshift-service-ca\") pod \"maas-api-6d669c7fcd-plj5l\" (UID: \"0ac7faef-5a37-4ac9-9513-958f0e4650ff\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-plj5l" Nov 02 01:21:31.651492 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:31.651343 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnrmf\" (UniqueName: \"kubernetes.io/projected/0ac7faef-5a37-4ac9-9513-958f0e4650ff-kube-api-access-dnrmf\") pod \"maas-api-6d669c7fcd-plj5l\" (UID: \"0ac7faef-5a37-4ac9-9513-958f0e4650ff\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-plj5l" Nov 02 01:21:31.651640 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:31.651615 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ac7faef-5a37-4ac9-9513-958f0e4650ff-maas-api-tls\") pod \"maas-api-6d669c7fcd-plj5l\" (UID: \"0ac7faef-5a37-4ac9-9513-958f0e4650ff\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-plj5l" Nov 02 01:21:31.652246 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:31.652222 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ac7faef-5a37-4ac9-9513-958f0e4650ff-openshift-service-ca\") pod \"maas-api-6d669c7fcd-plj5l\" (UID: \"0ac7faef-5a37-4ac9-9513-958f0e4650ff\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-plj5l" Nov 02 01:21:31.653888 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:31.653859 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ac7faef-5a37-4ac9-9513-958f0e4650ff-metrics-certs\") pod \"maas-api-6d669c7fcd-plj5l\" (UID: \"0ac7faef-5a37-4ac9-9513-958f0e4650ff\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-plj5l" Nov 02 01:21:31.654116 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:31.654095 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ac7faef-5a37-4ac9-9513-958f0e4650ff-maas-api-tls\") pod \"maas-api-6d669c7fcd-plj5l\" (UID: \"0ac7faef-5a37-4ac9-9513-958f0e4650ff\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-plj5l" Nov 02 01:21:31.663834 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:31.663811 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnrmf\" (UniqueName: \"kubernetes.io/projected/0ac7faef-5a37-4ac9-9513-958f0e4650ff-kube-api-access-dnrmf\") pod \"maas-api-6d669c7fcd-plj5l\" (UID: \"0ac7faef-5a37-4ac9-9513-958f0e4650ff\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-plj5l" Nov 02 01:21:31.747108 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:31.747004 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-plj5l" Nov 02 01:21:32.025862 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:32.025841 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-plj5l"] Nov 02 01:21:32.028109 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:21:32.028075 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ac7faef_5a37_4ac9_9513_958f0e4650ff.slice/crio-bd5fb7fcf44fce98ed35c3d78aa5ea80b034a25c6d5707d0998e877ad25fabd5 WatchSource:0}: Error finding container bd5fb7fcf44fce98ed35c3d78aa5ea80b034a25c6d5707d0998e877ad25fabd5: Status 404 returned error can't find the container with id bd5fb7fcf44fce98ed35c3d78aa5ea80b034a25c6d5707d0998e877ad25fabd5 Nov 02 01:21:32.162978 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:32.162940 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-plj5l" event={"ID":"0ac7faef-5a37-4ac9-9513-958f0e4650ff","Type":"ContainerStarted","Data":"bd5fb7fcf44fce98ed35c3d78aa5ea80b034a25c6d5707d0998e877ad25fabd5"} Nov 02 01:21:35.178356 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:35.178322 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-plj5l" event={"ID":"0ac7faef-5a37-4ac9-9513-958f0e4650ff","Type":"ContainerStarted","Data":"c166b0163475d008d065d0f0cf4562b334af972cd333c1692836ff6a6d99578d"} Nov 02 01:21:35.178777 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:35.178437 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-plj5l" Nov 02 01:21:35.198871 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:35.198819 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-plj5l" podStartSLOduration=1.509473772 podStartE2EDuration="4.198806419s" totalImagesPullingTime="2.689332647s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:21:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:21:35.198432057 +0000 UTC m=+399.819437268" watchObservedRunningTime="2026-11-02 01:21:35.198806419 +0000 UTC m=+399.819811629" Nov 02 01:21:41.188697 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:41.188627 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-plj5l" Nov 02 01:21:42.031398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:42.031328 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-n92cn"] Nov 02 01:21:42.034801 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:42.034777 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-n92cn" Nov 02 01:21:42.044440 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:42.044414 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-n92cn"] Nov 02 01:21:42.060784 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:42.060757 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 02 01:21:42.146750 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:42.146714 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2fqf\" (UniqueName: \"kubernetes.io/projected/57638221-b419-43d7-afee-05f96fe7bde9-kube-api-access-x2fqf\") pod \"authorino-6889cbd8c4-n92cn\" (UID: \"57638221-b419-43d7-afee-05f96fe7bde9\") " pod="kuadrant-system/authorino-6889cbd8c4-n92cn" Nov 02 01:21:42.146987 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:42.146855 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/57638221-b419-43d7-afee-05f96fe7bde9-tls-cert\") pod \"authorino-6889cbd8c4-n92cn\" (UID: \"57638221-b419-43d7-afee-05f96fe7bde9\") " pod="kuadrant-system/authorino-6889cbd8c4-n92cn" Nov 02 01:21:42.247946 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:42.247909 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/57638221-b419-43d7-afee-05f96fe7bde9-tls-cert\") pod \"authorino-6889cbd8c4-n92cn\" (UID: \"57638221-b419-43d7-afee-05f96fe7bde9\") " pod="kuadrant-system/authorino-6889cbd8c4-n92cn" Nov 02 01:21:42.247946 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:42.247953 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2fqf\" (UniqueName: \"kubernetes.io/projected/57638221-b419-43d7-afee-05f96fe7bde9-kube-api-access-x2fqf\") pod \"authorino-6889cbd8c4-n92cn\" (UID: \"57638221-b419-43d7-afee-05f96fe7bde9\") " pod="kuadrant-system/authorino-6889cbd8c4-n92cn" Nov 02 01:21:42.250523 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:42.250500 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/57638221-b419-43d7-afee-05f96fe7bde9-tls-cert\") pod \"authorino-6889cbd8c4-n92cn\" (UID: \"57638221-b419-43d7-afee-05f96fe7bde9\") " pod="kuadrant-system/authorino-6889cbd8c4-n92cn" Nov 02 01:21:42.260792 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:42.260762 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2fqf\" (UniqueName: \"kubernetes.io/projected/57638221-b419-43d7-afee-05f96fe7bde9-kube-api-access-x2fqf\") pod \"authorino-6889cbd8c4-n92cn\" (UID: \"57638221-b419-43d7-afee-05f96fe7bde9\") " pod="kuadrant-system/authorino-6889cbd8c4-n92cn" Nov 02 01:21:42.347802 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:42.347714 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-n92cn" Nov 02 01:21:42.348309 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:42.348286 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-n92cn"] Nov 02 01:21:42.620898 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:42.620862 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-n92cn"] Nov 02 01:21:42.624323 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:21:42.624286 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod57638221_b419_43d7_afee_05f96fe7bde9.slice/crio-c57066de45b9d03ca2c25c2df3f2226ef8026db9f9cf7dcc98d9e790eeb4ccb5 WatchSource:0}: Error finding container c57066de45b9d03ca2c25c2df3f2226ef8026db9f9cf7dcc98d9e790eeb4ccb5: Status 404 returned error can't find the container with id c57066de45b9d03ca2c25c2df3f2226ef8026db9f9cf7dcc98d9e790eeb4ccb5 Nov 02 01:21:43.213589 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:43.213557 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-n92cn" event={"ID":"57638221-b419-43d7-afee-05f96fe7bde9","Type":"ContainerStarted","Data":"c57066de45b9d03ca2c25c2df3f2226ef8026db9f9cf7dcc98d9e790eeb4ccb5"} Nov 02 01:21:44.219261 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:44.219223 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-n92cn" event={"ID":"57638221-b419-43d7-afee-05f96fe7bde9","Type":"ContainerStarted","Data":"b4841cfa14a3fa755b022b134e6f14cdfda59f88493ff96ff13116c603bf06ea"} Nov 02 01:21:44.219261 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:44.219234 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6889cbd8c4-n92cn" podUID="57638221-b419-43d7-afee-05f96fe7bde9" containerName="authorino" containerID="cri-o://b4841cfa14a3fa755b022b134e6f14cdfda59f88493ff96ff13116c603bf06ea" gracePeriod=30 Nov 02 01:21:44.475895 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:44.475835 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-n92cn" Nov 02 01:21:44.569193 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:44.569159 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57638221-b419-43d7-afee-05f96fe7bde9-kube-api-access-x2fqf\" (UniqueName: \"kubernetes.io/projected/57638221-b419-43d7-afee-05f96fe7bde9-kube-api-access-x2fqf\") pod \"57638221-b419-43d7-afee-05f96fe7bde9\" (UID: \"57638221-b419-43d7-afee-05f96fe7bde9\") " Nov 02 01:21:44.569363 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:44.569208 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57638221-b419-43d7-afee-05f96fe7bde9-tls-cert\" (UniqueName: \"kubernetes.io/secret/57638221-b419-43d7-afee-05f96fe7bde9-tls-cert\") pod \"57638221-b419-43d7-afee-05f96fe7bde9\" (UID: \"57638221-b419-43d7-afee-05f96fe7bde9\") " Nov 02 01:21:44.571541 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:44.571514 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57638221-b419-43d7-afee-05f96fe7bde9-kube-api-access-x2fqf" pod "57638221-b419-43d7-afee-05f96fe7bde9" (UID: "57638221-b419-43d7-afee-05f96fe7bde9"). InnerVolumeSpecName "kube-api-access-x2fqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:21:44.579697 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:44.579672 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57638221-b419-43d7-afee-05f96fe7bde9-tls-cert" pod "57638221-b419-43d7-afee-05f96fe7bde9" (UID: "57638221-b419-43d7-afee-05f96fe7bde9"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:21:44.670566 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:44.670533 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2fqf\" (UniqueName: \"kubernetes.io/projected/57638221-b419-43d7-afee-05f96fe7bde9-kube-api-access-x2fqf\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:21:44.670566 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:44.670564 4427 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/57638221-b419-43d7-afee-05f96fe7bde9-tls-cert\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:21:45.224738 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:45.224697 4427 generic.go:356] "Generic (PLEG): container finished" podID="57638221-b419-43d7-afee-05f96fe7bde9" containerID="b4841cfa14a3fa755b022b134e6f14cdfda59f88493ff96ff13116c603bf06ea" exitCode=0 Nov 02 01:21:45.224738 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:45.224740 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-n92cn" event={"ID":"57638221-b419-43d7-afee-05f96fe7bde9","Type":"ContainerDied","Data":"b4841cfa14a3fa755b022b134e6f14cdfda59f88493ff96ff13116c603bf06ea"} Nov 02 01:21:45.225239 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:45.224741 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-n92cn" Nov 02 01:21:45.225239 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:45.224770 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-n92cn" event={"ID":"57638221-b419-43d7-afee-05f96fe7bde9","Type":"ContainerDied","Data":"c57066de45b9d03ca2c25c2df3f2226ef8026db9f9cf7dcc98d9e790eeb4ccb5"} Nov 02 01:21:45.225239 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:45.224793 4427 scope.go:122] "RemoveContainer" containerID="b4841cfa14a3fa755b022b134e6f14cdfda59f88493ff96ff13116c603bf06ea" Nov 02 01:21:45.233859 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:45.233841 4427 scope.go:122] "RemoveContainer" containerID="b4841cfa14a3fa755b022b134e6f14cdfda59f88493ff96ff13116c603bf06ea" Nov 02 01:21:45.234159 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:21:45.234139 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4841cfa14a3fa755b022b134e6f14cdfda59f88493ff96ff13116c603bf06ea\": container with ID starting with b4841cfa14a3fa755b022b134e6f14cdfda59f88493ff96ff13116c603bf06ea not found: ID does not exist" containerID="b4841cfa14a3fa755b022b134e6f14cdfda59f88493ff96ff13116c603bf06ea" Nov 02 01:21:45.234202 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:45.234170 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4841cfa14a3fa755b022b134e6f14cdfda59f88493ff96ff13116c603bf06ea"} err="failed to get container status \"b4841cfa14a3fa755b022b134e6f14cdfda59f88493ff96ff13116c603bf06ea\": rpc error: code = NotFound desc = could not find container \"b4841cfa14a3fa755b022b134e6f14cdfda59f88493ff96ff13116c603bf06ea\": container with ID starting with b4841cfa14a3fa755b022b134e6f14cdfda59f88493ff96ff13116c603bf06ea not found: ID does not exist" Nov 02 01:21:45.246575 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:45.246546 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-n92cn"] Nov 02 01:21:45.250018 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:45.249998 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-n92cn"] Nov 02 01:21:45.536848 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:45.536766 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57638221-b419-43d7-afee-05f96fe7bde9" path="/var/lib/kubelet/pods/57638221-b419-43d7-afee-05f96fe7bde9/volumes" Nov 02 01:21:51.118559 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:51.118512 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xh22d" Nov 02 01:21:51.609266 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:51.609234 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-hhbp4"] Nov 02 01:21:51.609583 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:51.609513 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-hhbp4" podUID="5204ad71-2e13-4225-90fd-9ee6c34831cf" containerName="authorino" containerID="cri-o://8ce95b2151863cc80791db9f87c53bc33e4df496aeaa26f41222f5b8b6b5e68d" gracePeriod=30 Nov 02 01:21:51.730322 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:51.730292 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-plj5l"] Nov 02 01:21:51.730522 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:51.730501 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-plj5l" podUID="0ac7faef-5a37-4ac9-9513-958f0e4650ff" containerName="maas-api" containerID="cri-o://c166b0163475d008d065d0f0cf4562b334af972cd333c1692836ff6a6d99578d" gracePeriod=30 Nov 02 01:21:51.859347 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:51.859278 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-hhbp4" Nov 02 01:21:51.985744 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:51.985712 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-plj5l" Nov 02 01:21:52.032922 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:52.032882 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5204ad71-2e13-4225-90fd-9ee6c34831cf-kube-api-access-8tp8r\" (UniqueName: \"kubernetes.io/projected/5204ad71-2e13-4225-90fd-9ee6c34831cf-kube-api-access-8tp8r\") pod \"5204ad71-2e13-4225-90fd-9ee6c34831cf\" (UID: \"5204ad71-2e13-4225-90fd-9ee6c34831cf\") " Nov 02 01:21:52.035085 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:52.035051 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5204ad71-2e13-4225-90fd-9ee6c34831cf-kube-api-access-8tp8r" pod "5204ad71-2e13-4225-90fd-9ee6c34831cf" (UID: "5204ad71-2e13-4225-90fd-9ee6c34831cf"). InnerVolumeSpecName "kube-api-access-8tp8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:21:52.133499 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:52.133403 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ac7faef-5a37-4ac9-9513-958f0e4650ff-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ac7faef-5a37-4ac9-9513-958f0e4650ff-metrics-certs\") pod \"0ac7faef-5a37-4ac9-9513-958f0e4650ff\" (UID: \"0ac7faef-5a37-4ac9-9513-958f0e4650ff\") " Nov 02 01:21:52.133499 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:52.133449 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ac7faef-5a37-4ac9-9513-958f0e4650ff-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ac7faef-5a37-4ac9-9513-958f0e4650ff-maas-api-tls\") pod \"0ac7faef-5a37-4ac9-9513-958f0e4650ff\" (UID: \"0ac7faef-5a37-4ac9-9513-958f0e4650ff\") " Nov 02 01:21:52.134001 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:52.133516 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ac7faef-5a37-4ac9-9513-958f0e4650ff-kube-api-access-dnrmf\" (UniqueName: \"kubernetes.io/projected/0ac7faef-5a37-4ac9-9513-958f0e4650ff-kube-api-access-dnrmf\") pod \"0ac7faef-5a37-4ac9-9513-958f0e4650ff\" (UID: \"0ac7faef-5a37-4ac9-9513-958f0e4650ff\") " Nov 02 01:21:52.134001 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:52.133579 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ac7faef-5a37-4ac9-9513-958f0e4650ff-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ac7faef-5a37-4ac9-9513-958f0e4650ff-openshift-service-ca\") pod \"0ac7faef-5a37-4ac9-9513-958f0e4650ff\" (UID: \"0ac7faef-5a37-4ac9-9513-958f0e4650ff\") " Nov 02 01:21:52.134001 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:52.133739 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tp8r\" (UniqueName: \"kubernetes.io/projected/5204ad71-2e13-4225-90fd-9ee6c34831cf-kube-api-access-8tp8r\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:21:52.134109 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:52.134034 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ac7faef-5a37-4ac9-9513-958f0e4650ff-openshift-service-ca" pod "0ac7faef-5a37-4ac9-9513-958f0e4650ff" (UID: "0ac7faef-5a37-4ac9-9513-958f0e4650ff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:21:52.135900 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:52.135877 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ac7faef-5a37-4ac9-9513-958f0e4650ff-kube-api-access-dnrmf" pod "0ac7faef-5a37-4ac9-9513-958f0e4650ff" (UID: "0ac7faef-5a37-4ac9-9513-958f0e4650ff"). InnerVolumeSpecName "kube-api-access-dnrmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:21:52.135975 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:52.135897 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ac7faef-5a37-4ac9-9513-958f0e4650ff-metrics-certs" pod "0ac7faef-5a37-4ac9-9513-958f0e4650ff" (UID: "0ac7faef-5a37-4ac9-9513-958f0e4650ff"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:21:52.135975 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:52.135880 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ac7faef-5a37-4ac9-9513-958f0e4650ff-maas-api-tls" pod "0ac7faef-5a37-4ac9-9513-958f0e4650ff" (UID: "0ac7faef-5a37-4ac9-9513-958f0e4650ff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:21:52.235154 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:52.235117 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnrmf\" (UniqueName: \"kubernetes.io/projected/0ac7faef-5a37-4ac9-9513-958f0e4650ff-kube-api-access-dnrmf\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:21:52.235154 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:52.235146 4427 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ac7faef-5a37-4ac9-9513-958f0e4650ff-openshift-service-ca\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:21:52.235154 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:52.235156 4427 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ac7faef-5a37-4ac9-9513-958f0e4650ff-metrics-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:21:52.235399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:52.235168 4427 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ac7faef-5a37-4ac9-9513-958f0e4650ff-maas-api-tls\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:21:52.255881 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:52.255853 4427 generic.go:356] "Generic (PLEG): container finished" podID="5204ad71-2e13-4225-90fd-9ee6c34831cf" containerID="8ce95b2151863cc80791db9f87c53bc33e4df496aeaa26f41222f5b8b6b5e68d" exitCode=0 Nov 02 01:21:52.256044 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:52.255893 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-hhbp4" Nov 02 01:21:52.256044 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:52.255931 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-hhbp4" event={"ID":"5204ad71-2e13-4225-90fd-9ee6c34831cf","Type":"ContainerDied","Data":"8ce95b2151863cc80791db9f87c53bc33e4df496aeaa26f41222f5b8b6b5e68d"} Nov 02 01:21:52.256044 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:52.255971 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-hhbp4" event={"ID":"5204ad71-2e13-4225-90fd-9ee6c34831cf","Type":"ContainerDied","Data":"ec4d73fd82f8c6d2f0a384398be948d0a90a199a495ef864c4ff65eb9bdffb5c"} Nov 02 01:21:52.256044 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:52.255993 4427 scope.go:122] "RemoveContainer" containerID="8ce95b2151863cc80791db9f87c53bc33e4df496aeaa26f41222f5b8b6b5e68d" Nov 02 01:21:52.257516 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:52.257493 4427 generic.go:356] "Generic (PLEG): container finished" podID="0ac7faef-5a37-4ac9-9513-958f0e4650ff" containerID="c166b0163475d008d065d0f0cf4562b334af972cd333c1692836ff6a6d99578d" exitCode=0 Nov 02 01:21:52.257621 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:52.257531 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-plj5l" Nov 02 01:21:52.257621 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:52.257573 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-plj5l" event={"ID":"0ac7faef-5a37-4ac9-9513-958f0e4650ff","Type":"ContainerDied","Data":"c166b0163475d008d065d0f0cf4562b334af972cd333c1692836ff6a6d99578d"} Nov 02 01:21:52.257621 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:52.257615 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-plj5l" event={"ID":"0ac7faef-5a37-4ac9-9513-958f0e4650ff","Type":"ContainerDied","Data":"bd5fb7fcf44fce98ed35c3d78aa5ea80b034a25c6d5707d0998e877ad25fabd5"} Nov 02 01:21:52.268041 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:52.267873 4427 scope.go:122] "RemoveContainer" containerID="8ce95b2151863cc80791db9f87c53bc33e4df496aeaa26f41222f5b8b6b5e68d" Nov 02 01:21:52.268198 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:21:52.268168 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ce95b2151863cc80791db9f87c53bc33e4df496aeaa26f41222f5b8b6b5e68d\": container with ID starting with 8ce95b2151863cc80791db9f87c53bc33e4df496aeaa26f41222f5b8b6b5e68d not found: ID does not exist" containerID="8ce95b2151863cc80791db9f87c53bc33e4df496aeaa26f41222f5b8b6b5e68d" Nov 02 01:21:52.268367 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:52.268206 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ce95b2151863cc80791db9f87c53bc33e4df496aeaa26f41222f5b8b6b5e68d"} err="failed to get container status \"8ce95b2151863cc80791db9f87c53bc33e4df496aeaa26f41222f5b8b6b5e68d\": rpc error: code = NotFound desc = could not find container \"8ce95b2151863cc80791db9f87c53bc33e4df496aeaa26f41222f5b8b6b5e68d\": container with ID starting with 8ce95b2151863cc80791db9f87c53bc33e4df496aeaa26f41222f5b8b6b5e68d not found: ID does not exist" Nov 02 01:21:52.268367 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:52.268230 4427 scope.go:122] "RemoveContainer" containerID="c166b0163475d008d065d0f0cf4562b334af972cd333c1692836ff6a6d99578d" Nov 02 01:21:52.277531 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:52.277512 4427 scope.go:122] "RemoveContainer" containerID="c166b0163475d008d065d0f0cf4562b334af972cd333c1692836ff6a6d99578d" Nov 02 01:21:52.277829 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:21:52.277810 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c166b0163475d008d065d0f0cf4562b334af972cd333c1692836ff6a6d99578d\": container with ID starting with c166b0163475d008d065d0f0cf4562b334af972cd333c1692836ff6a6d99578d not found: ID does not exist" containerID="c166b0163475d008d065d0f0cf4562b334af972cd333c1692836ff6a6d99578d" Nov 02 01:21:52.277875 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:52.277838 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c166b0163475d008d065d0f0cf4562b334af972cd333c1692836ff6a6d99578d"} err="failed to get container status \"c166b0163475d008d065d0f0cf4562b334af972cd333c1692836ff6a6d99578d\": rpc error: code = NotFound desc = could not find container \"c166b0163475d008d065d0f0cf4562b334af972cd333c1692836ff6a6d99578d\": container with ID starting with c166b0163475d008d065d0f0cf4562b334af972cd333c1692836ff6a6d99578d not found: ID does not exist" Nov 02 01:21:52.284886 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:52.284865 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-plj5l"] Nov 02 01:21:52.288952 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:52.288934 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-plj5l"] Nov 02 01:21:52.299640 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:52.299617 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-hhbp4"] Nov 02 01:21:52.303261 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:52.303243 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-hhbp4"] Nov 02 01:21:53.531626 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:53.531595 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ac7faef-5a37-4ac9-9513-958f0e4650ff" path="/var/lib/kubelet/pods/0ac7faef-5a37-4ac9-9513-958f0e4650ff/volumes" Nov 02 01:21:53.532019 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:21:53.531906 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5204ad71-2e13-4225-90fd-9ee6c34831cf" path="/var/lib/kubelet/pods/5204ad71-2e13-4225-90fd-9ee6c34831cf/volumes" Nov 02 01:22:01.581166 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:22:01.581129 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:22:12.677736 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:22:12.677703 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-rrmfw"] Nov 02 01:22:12.678128 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:22:12.678050 4427 cpu_manager.go:414] "containerMap: removing container" podUID="5204ad71-2e13-4225-90fd-9ee6c34831cf" containerName="authorino" Nov 02 01:22:12.678128 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:22:12.678061 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5204ad71-2e13-4225-90fd-9ee6c34831cf" containerName="authorino" Nov 02 01:22:12.678128 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:22:12.678072 4427 cpu_manager.go:414] "containerMap: removing container" podUID="0ac7faef-5a37-4ac9-9513-958f0e4650ff" containerName="maas-api" Nov 02 01:22:12.678128 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:22:12.678077 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ac7faef-5a37-4ac9-9513-958f0e4650ff" containerName="maas-api" Nov 02 01:22:12.678128 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:22:12.678099 4427 cpu_manager.go:414] "containerMap: removing container" podUID="57638221-b419-43d7-afee-05f96fe7bde9" containerName="authorino" Nov 02 01:22:12.678128 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:22:12.678104 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57638221-b419-43d7-afee-05f96fe7bde9" containerName="authorino" Nov 02 01:22:12.678305 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:22:12.678161 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="57638221-b419-43d7-afee-05f96fe7bde9" containerName="authorino" Nov 02 01:22:12.678305 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:22:12.678172 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="5204ad71-2e13-4225-90fd-9ee6c34831cf" containerName="authorino" Nov 02 01:22:12.678305 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:22:12.678178 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ac7faef-5a37-4ac9-9513-958f0e4650ff" containerName="maas-api" Nov 02 01:22:12.681211 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:22:12.681195 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rrmfw" Nov 02 01:22:12.691786 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:22:12.691762 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-rrmfw"] Nov 02 01:22:12.698133 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:22:12.698108 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfxp6\" (UniqueName: \"kubernetes.io/projected/6e1cb75b-5ed8-4213-a5db-4f60848c2612-kube-api-access-wfxp6\") pod \"llmisvc-controller-manager-6466c554d8-rrmfw\" (UID: \"6e1cb75b-5ed8-4213-a5db-4f60848c2612\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-rrmfw" Nov 02 01:22:12.698240 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:22:12.698145 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6e1cb75b-5ed8-4213-a5db-4f60848c2612-cert\") pod \"llmisvc-controller-manager-6466c554d8-rrmfw\" (UID: \"6e1cb75b-5ed8-4213-a5db-4f60848c2612\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-rrmfw" Nov 02 01:22:12.798633 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:22:12.798591 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfxp6\" (UniqueName: \"kubernetes.io/projected/6e1cb75b-5ed8-4213-a5db-4f60848c2612-kube-api-access-wfxp6\") pod \"llmisvc-controller-manager-6466c554d8-rrmfw\" (UID: \"6e1cb75b-5ed8-4213-a5db-4f60848c2612\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-rrmfw" Nov 02 01:22:12.798633 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:22:12.798640 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6e1cb75b-5ed8-4213-a5db-4f60848c2612-cert\") pod \"llmisvc-controller-manager-6466c554d8-rrmfw\" (UID: \"6e1cb75b-5ed8-4213-a5db-4f60848c2612\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-rrmfw" Nov 02 01:22:12.801287 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:22:12.801262 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6e1cb75b-5ed8-4213-a5db-4f60848c2612-cert\") pod \"llmisvc-controller-manager-6466c554d8-rrmfw\" (UID: \"6e1cb75b-5ed8-4213-a5db-4f60848c2612\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-rrmfw" Nov 02 01:22:12.811363 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:22:12.811339 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfxp6\" (UniqueName: \"kubernetes.io/projected/6e1cb75b-5ed8-4213-a5db-4f60848c2612-kube-api-access-wfxp6\") pod \"llmisvc-controller-manager-6466c554d8-rrmfw\" (UID: \"6e1cb75b-5ed8-4213-a5db-4f60848c2612\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-rrmfw" Nov 02 01:22:12.994349 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:22:12.994257 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rrmfw" Nov 02 01:22:13.269781 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:22:13.269171 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-rrmfw"] Nov 02 01:22:13.348960 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:22:13.348913 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rrmfw" event={"ID":"6e1cb75b-5ed8-4213-a5db-4f60848c2612","Type":"ContainerStarted","Data":"88f6e78ccf4b7d480f7de371304628b8e0e024225b679e75418907e675989dc3"} Nov 02 01:22:14.355053 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:22:14.354955 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rrmfw" event={"ID":"6e1cb75b-5ed8-4213-a5db-4f60848c2612","Type":"ContainerStarted","Data":"2caf3f82884fb6070bdea3f0d331c1b1fc281ccbba9169fecc6564fb68acb761"} Nov 02 01:22:14.355470 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:22:14.355081 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rrmfw" Nov 02 01:22:14.374899 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:22:14.374848 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rrmfw" podStartSLOduration=1.7781742010000001 podStartE2EDuration="2.374833063s" totalImagesPullingTime="596.658862ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:22:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:22:14.373218233 +0000 UTC m=+438.994223443" watchObservedRunningTime="2026-11-02 01:22:14.374833063 +0000 UTC m=+438.995838273" Nov 02 01:22:31.581457 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:22:31.581414 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:22:45.361943 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:22:45.361903 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rrmfw" Nov 02 01:22:45.416396 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:22:45.416340 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-xh22d"] Nov 02 01:22:45.416664 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:22:45.416617 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xh22d" podUID="b8c9ab8f-5d15-497f-9365-b72bdbb46cc9" containerName="manager" containerID="cri-o://515ebc453f8f9d1772dfd5b2b0287b9daef0a0fdab332d6c4a75766b6f5e1088" gracePeriod=30 Nov 02 01:22:45.654605 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:22:45.654581 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xh22d" Nov 02 01:22:45.784161 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:22:45.784119 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8c9ab8f-5d15-497f-9365-b72bdbb46cc9-kube-api-access-b2txq\" (UniqueName: \"kubernetes.io/projected/b8c9ab8f-5d15-497f-9365-b72bdbb46cc9-kube-api-access-b2txq\") pod \"b8c9ab8f-5d15-497f-9365-b72bdbb46cc9\" (UID: \"b8c9ab8f-5d15-497f-9365-b72bdbb46cc9\") " Nov 02 01:22:45.784161 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:22:45.784164 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8c9ab8f-5d15-497f-9365-b72bdbb46cc9-cert\" (UniqueName: \"kubernetes.io/secret/b8c9ab8f-5d15-497f-9365-b72bdbb46cc9-cert\") pod \"b8c9ab8f-5d15-497f-9365-b72bdbb46cc9\" (UID: \"b8c9ab8f-5d15-497f-9365-b72bdbb46cc9\") " Nov 02 01:22:45.786555 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:22:45.786518 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8c9ab8f-5d15-497f-9365-b72bdbb46cc9-kube-api-access-b2txq" pod "b8c9ab8f-5d15-497f-9365-b72bdbb46cc9" (UID: "b8c9ab8f-5d15-497f-9365-b72bdbb46cc9"). InnerVolumeSpecName "kube-api-access-b2txq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:22:45.786657 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:22:45.786610 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8c9ab8f-5d15-497f-9365-b72bdbb46cc9-cert" pod "b8c9ab8f-5d15-497f-9365-b72bdbb46cc9" (UID: "b8c9ab8f-5d15-497f-9365-b72bdbb46cc9"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:22:45.885037 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:22:45.884993 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2txq\" (UniqueName: \"kubernetes.io/projected/b8c9ab8f-5d15-497f-9365-b72bdbb46cc9-kube-api-access-b2txq\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:22:45.885037 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:22:45.885027 4427 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b8c9ab8f-5d15-497f-9365-b72bdbb46cc9-cert\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:22:46.491182 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:22:46.491150 4427 generic.go:356] "Generic (PLEG): container finished" podID="b8c9ab8f-5d15-497f-9365-b72bdbb46cc9" containerID="515ebc453f8f9d1772dfd5b2b0287b9daef0a0fdab332d6c4a75766b6f5e1088" exitCode=0 Nov 02 01:22:46.491640 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:22:46.491199 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xh22d" Nov 02 01:22:46.491640 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:22:46.491243 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xh22d" event={"ID":"b8c9ab8f-5d15-497f-9365-b72bdbb46cc9","Type":"ContainerDied","Data":"515ebc453f8f9d1772dfd5b2b0287b9daef0a0fdab332d6c4a75766b6f5e1088"} Nov 02 01:22:46.491640 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:22:46.491293 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xh22d" event={"ID":"b8c9ab8f-5d15-497f-9365-b72bdbb46cc9","Type":"ContainerDied","Data":"af71bfcc05553dc8b966e4594cdb64cb2cd1435609d73d3d0fe50b755702e801"} Nov 02 01:22:46.491640 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:22:46.491315 4427 scope.go:122] "RemoveContainer" containerID="515ebc453f8f9d1772dfd5b2b0287b9daef0a0fdab332d6c4a75766b6f5e1088" Nov 02 01:22:46.499739 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:22:46.499724 4427 scope.go:122] "RemoveContainer" containerID="515ebc453f8f9d1772dfd5b2b0287b9daef0a0fdab332d6c4a75766b6f5e1088" Nov 02 01:22:46.499999 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:22:46.499983 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"515ebc453f8f9d1772dfd5b2b0287b9daef0a0fdab332d6c4a75766b6f5e1088\": container with ID starting with 515ebc453f8f9d1772dfd5b2b0287b9daef0a0fdab332d6c4a75766b6f5e1088 not found: ID does not exist" containerID="515ebc453f8f9d1772dfd5b2b0287b9daef0a0fdab332d6c4a75766b6f5e1088" Nov 02 01:22:46.500038 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:22:46.500008 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"515ebc453f8f9d1772dfd5b2b0287b9daef0a0fdab332d6c4a75766b6f5e1088"} err="failed to get container status \"515ebc453f8f9d1772dfd5b2b0287b9daef0a0fdab332d6c4a75766b6f5e1088\": rpc error: code = NotFound desc = could not find container \"515ebc453f8f9d1772dfd5b2b0287b9daef0a0fdab332d6c4a75766b6f5e1088\": container with ID starting with 515ebc453f8f9d1772dfd5b2b0287b9daef0a0fdab332d6c4a75766b6f5e1088 not found: ID does not exist" Nov 02 01:22:46.511877 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:22:46.511848 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-xh22d"] Nov 02 01:22:46.515988 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:22:46.515967 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-xh22d"] Nov 02 01:22:47.532070 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:22:47.532041 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8c9ab8f-5d15-497f-9365-b72bdbb46cc9" path="/var/lib/kubelet/pods/b8c9ab8f-5d15-497f-9365-b72bdbb46cc9/volumes" Nov 02 01:23:01.580602 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:23:01.580566 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:23:11.134486 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:11.134397 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66"] Nov 02 01:23:11.135042 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:11.135010 4427 cpu_manager.go:414] "containerMap: removing container" podUID="b8c9ab8f-5d15-497f-9365-b72bdbb46cc9" containerName="manager" Nov 02 01:23:11.135042 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:11.135031 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8c9ab8f-5d15-497f-9365-b72bdbb46cc9" containerName="manager" Nov 02 01:23:11.135196 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:11.135169 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8c9ab8f-5d15-497f-9365-b72bdbb46cc9" containerName="manager" Nov 02 01:23:11.138067 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:11.138046 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" Nov 02 01:23:11.143334 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:11.143307 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 02 01:23:11.144708 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:11.144687 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 02 01:23:11.144811 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:11.144800 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Nov 02 01:23:11.144860 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:11.144849 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-5855b\"" Nov 02 01:23:11.151354 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:11.151330 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66"] Nov 02 01:23:11.187276 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:11.187241 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d73ae484-a9b8-4565-a346-0d803a1981ae-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66\" (UID: \"d73ae484-a9b8-4565-a346-0d803a1981ae\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" Nov 02 01:23:11.187478 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:11.187300 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d73ae484-a9b8-4565-a346-0d803a1981ae-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66\" (UID: \"d73ae484-a9b8-4565-a346-0d803a1981ae\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" Nov 02 01:23:11.187478 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:11.187469 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d73ae484-a9b8-4565-a346-0d803a1981ae-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66\" (UID: \"d73ae484-a9b8-4565-a346-0d803a1981ae\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" Nov 02 01:23:11.187557 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:11.187510 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vwwm\" (UniqueName: \"kubernetes.io/projected/d73ae484-a9b8-4565-a346-0d803a1981ae-kube-api-access-2vwwm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66\" (UID: \"d73ae484-a9b8-4565-a346-0d803a1981ae\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" Nov 02 01:23:11.187591 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:11.187559 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d73ae484-a9b8-4565-a346-0d803a1981ae-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66\" (UID: \"d73ae484-a9b8-4565-a346-0d803a1981ae\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" Nov 02 01:23:11.187692 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:11.187668 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d73ae484-a9b8-4565-a346-0d803a1981ae-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66\" (UID: \"d73ae484-a9b8-4565-a346-0d803a1981ae\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" Nov 02 01:23:11.288396 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:11.288335 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d73ae484-a9b8-4565-a346-0d803a1981ae-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66\" (UID: \"d73ae484-a9b8-4565-a346-0d803a1981ae\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" Nov 02 01:23:11.288581 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:11.288414 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d73ae484-a9b8-4565-a346-0d803a1981ae-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66\" (UID: \"d73ae484-a9b8-4565-a346-0d803a1981ae\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" Nov 02 01:23:11.288581 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:11.288466 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d73ae484-a9b8-4565-a346-0d803a1981ae-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66\" (UID: \"d73ae484-a9b8-4565-a346-0d803a1981ae\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" Nov 02 01:23:11.288581 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:11.288507 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d73ae484-a9b8-4565-a346-0d803a1981ae-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66\" (UID: \"d73ae484-a9b8-4565-a346-0d803a1981ae\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" Nov 02 01:23:11.288581 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:11.288552 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d73ae484-a9b8-4565-a346-0d803a1981ae-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66\" (UID: \"d73ae484-a9b8-4565-a346-0d803a1981ae\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" Nov 02 01:23:11.288812 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:11.288594 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vwwm\" (UniqueName: \"kubernetes.io/projected/d73ae484-a9b8-4565-a346-0d803a1981ae-kube-api-access-2vwwm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66\" (UID: \"d73ae484-a9b8-4565-a346-0d803a1981ae\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" Nov 02 01:23:11.288914 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:11.288887 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d73ae484-a9b8-4565-a346-0d803a1981ae-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66\" (UID: \"d73ae484-a9b8-4565-a346-0d803a1981ae\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" Nov 02 01:23:11.288978 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:11.288943 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d73ae484-a9b8-4565-a346-0d803a1981ae-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66\" (UID: \"d73ae484-a9b8-4565-a346-0d803a1981ae\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" Nov 02 01:23:11.289025 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:11.288976 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d73ae484-a9b8-4565-a346-0d803a1981ae-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66\" (UID: \"d73ae484-a9b8-4565-a346-0d803a1981ae\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" Nov 02 01:23:11.290932 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:11.290900 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d73ae484-a9b8-4565-a346-0d803a1981ae-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66\" (UID: \"d73ae484-a9b8-4565-a346-0d803a1981ae\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" Nov 02 01:23:11.291164 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:11.291143 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d73ae484-a9b8-4565-a346-0d803a1981ae-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66\" (UID: \"d73ae484-a9b8-4565-a346-0d803a1981ae\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" Nov 02 01:23:11.307693 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:11.307668 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vwwm\" (UniqueName: \"kubernetes.io/projected/d73ae484-a9b8-4565-a346-0d803a1981ae-kube-api-access-2vwwm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66\" (UID: \"d73ae484-a9b8-4565-a346-0d803a1981ae\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" Nov 02 01:23:11.451321 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:11.451236 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" Nov 02 01:23:11.730823 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:11.730797 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66"] Nov 02 01:23:11.733110 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:23:11.733077 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd73ae484_a9b8_4565_a346_0d803a1981ae.slice/crio-951a89a67f1bd9035b982877f442f581f0f27dd06c7cb04378ff99300056c1c5 WatchSource:0}: Error finding container 951a89a67f1bd9035b982877f442f581f0f27dd06c7cb04378ff99300056c1c5: Status 404 returned error can't find the container with id 951a89a67f1bd9035b982877f442f581f0f27dd06c7cb04378ff99300056c1c5 Nov 02 01:23:12.609858 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:12.609818 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" event={"ID":"d73ae484-a9b8-4565-a346-0d803a1981ae","Type":"ContainerStarted","Data":"951a89a67f1bd9035b982877f442f581f0f27dd06c7cb04378ff99300056c1c5"} Nov 02 01:23:13.615847 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:13.615811 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" event={"ID":"d73ae484-a9b8-4565-a346-0d803a1981ae","Type":"ContainerStarted","Data":"32d357446a3a01cd3e2cda3632ac0b86f83087f164baef1ac0e48bfd1943512a"} Nov 02 01:23:13.636927 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:13.636874 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" podStartSLOduration=1.427313692 podStartE2EDuration="2.636861089s" totalImagesPullingTime="1.209547397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:23:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:23:13.635365144 +0000 UTC m=+498.256370355" watchObservedRunningTime="2026-11-02 01:23:13.636861089 +0000 UTC m=+498.257866299" Nov 02 01:23:14.452311 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:14.452271 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" Nov 02 01:23:21.452113 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:21.452077 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" Nov 02 01:23:21.464701 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:21.464677 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" Nov 02 01:23:21.670802 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:21.670760 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" Nov 02 01:23:23.181284 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:23.181249 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fm4lg"] Nov 02 01:23:23.184978 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:23.184949 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fm4lg" Nov 02 01:23:23.191623 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:23.191597 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 02 01:23:23.202841 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:23.202811 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fm4lg"] Nov 02 01:23:23.275495 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:23.275461 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-48jtq\"" Nov 02 01:23:23.295628 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:23.295593 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99cc3c61-f501-4a64-8f7e-0f41058aac68-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-fm4lg\" (UID: \"99cc3c61-f501-4a64-8f7e-0f41058aac68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fm4lg" Nov 02 01:23:23.295782 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:23.295726 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99cc3c61-f501-4a64-8f7e-0f41058aac68-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-fm4lg\" (UID: \"99cc3c61-f501-4a64-8f7e-0f41058aac68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fm4lg" Nov 02 01:23:23.295782 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:23.295763 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99cc3c61-f501-4a64-8f7e-0f41058aac68-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-fm4lg\" (UID: \"99cc3c61-f501-4a64-8f7e-0f41058aac68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fm4lg" Nov 02 01:23:23.295874 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:23.295792 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99cc3c61-f501-4a64-8f7e-0f41058aac68-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-fm4lg\" (UID: \"99cc3c61-f501-4a64-8f7e-0f41058aac68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fm4lg" Nov 02 01:23:23.295911 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:23.295865 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99cc3c61-f501-4a64-8f7e-0f41058aac68-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-fm4lg\" (UID: \"99cc3c61-f501-4a64-8f7e-0f41058aac68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fm4lg" Nov 02 01:23:23.295945 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:23.295907 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rw4m\" (UniqueName: \"kubernetes.io/projected/99cc3c61-f501-4a64-8f7e-0f41058aac68-kube-api-access-5rw4m\") pod \"maas-default-gateway-openshift-default-76597d6855-fm4lg\" (UID: \"99cc3c61-f501-4a64-8f7e-0f41058aac68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fm4lg" Nov 02 01:23:23.295994 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:23.295972 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99cc3c61-f501-4a64-8f7e-0f41058aac68-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-fm4lg\" (UID: \"99cc3c61-f501-4a64-8f7e-0f41058aac68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fm4lg" Nov 02 01:23:23.296080 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:23.296064 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99cc3c61-f501-4a64-8f7e-0f41058aac68-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-fm4lg\" (UID: \"99cc3c61-f501-4a64-8f7e-0f41058aac68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fm4lg" Nov 02 01:23:23.296156 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:23.296141 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99cc3c61-f501-4a64-8f7e-0f41058aac68-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-fm4lg\" (UID: \"99cc3c61-f501-4a64-8f7e-0f41058aac68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fm4lg" Nov 02 01:23:23.396966 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:23.396922 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99cc3c61-f501-4a64-8f7e-0f41058aac68-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-fm4lg\" (UID: \"99cc3c61-f501-4a64-8f7e-0f41058aac68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fm4lg" Nov 02 01:23:23.397153 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:23.396977 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99cc3c61-f501-4a64-8f7e-0f41058aac68-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-fm4lg\" (UID: \"99cc3c61-f501-4a64-8f7e-0f41058aac68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fm4lg" Nov 02 01:23:23.397153 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:23.397006 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99cc3c61-f501-4a64-8f7e-0f41058aac68-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-fm4lg\" (UID: \"99cc3c61-f501-4a64-8f7e-0f41058aac68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fm4lg" Nov 02 01:23:23.397153 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:23.397030 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rw4m\" (UniqueName: \"kubernetes.io/projected/99cc3c61-f501-4a64-8f7e-0f41058aac68-kube-api-access-5rw4m\") pod \"maas-default-gateway-openshift-default-76597d6855-fm4lg\" (UID: \"99cc3c61-f501-4a64-8f7e-0f41058aac68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fm4lg" Nov 02 01:23:23.397364 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:23.397182 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99cc3c61-f501-4a64-8f7e-0f41058aac68-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-fm4lg\" (UID: \"99cc3c61-f501-4a64-8f7e-0f41058aac68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fm4lg" Nov 02 01:23:23.397364 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:23.397245 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99cc3c61-f501-4a64-8f7e-0f41058aac68-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-fm4lg\" (UID: \"99cc3c61-f501-4a64-8f7e-0f41058aac68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fm4lg" Nov 02 01:23:23.397364 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:23.397327 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99cc3c61-f501-4a64-8f7e-0f41058aac68-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-fm4lg\" (UID: \"99cc3c61-f501-4a64-8f7e-0f41058aac68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fm4lg" Nov 02 01:23:23.397557 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:23.397366 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99cc3c61-f501-4a64-8f7e-0f41058aac68-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-fm4lg\" (UID: \"99cc3c61-f501-4a64-8f7e-0f41058aac68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fm4lg" Nov 02 01:23:23.397557 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:23.397464 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99cc3c61-f501-4a64-8f7e-0f41058aac68-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-fm4lg\" (UID: \"99cc3c61-f501-4a64-8f7e-0f41058aac68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fm4lg" Nov 02 01:23:23.397557 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:23.397551 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99cc3c61-f501-4a64-8f7e-0f41058aac68-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-fm4lg\" (UID: \"99cc3c61-f501-4a64-8f7e-0f41058aac68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fm4lg" Nov 02 01:23:23.397832 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:23.397809 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99cc3c61-f501-4a64-8f7e-0f41058aac68-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-fm4lg\" (UID: \"99cc3c61-f501-4a64-8f7e-0f41058aac68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fm4lg" Nov 02 01:23:23.397996 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:23.397965 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99cc3c61-f501-4a64-8f7e-0f41058aac68-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-fm4lg\" (UID: \"99cc3c61-f501-4a64-8f7e-0f41058aac68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fm4lg" Nov 02 01:23:23.398113 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:23.398085 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99cc3c61-f501-4a64-8f7e-0f41058aac68-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-fm4lg\" (UID: \"99cc3c61-f501-4a64-8f7e-0f41058aac68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fm4lg" Nov 02 01:23:23.398182 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:23.398144 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99cc3c61-f501-4a64-8f7e-0f41058aac68-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-fm4lg\" (UID: \"99cc3c61-f501-4a64-8f7e-0f41058aac68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fm4lg" Nov 02 01:23:23.399603 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:23.399581 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99cc3c61-f501-4a64-8f7e-0f41058aac68-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-fm4lg\" (UID: \"99cc3c61-f501-4a64-8f7e-0f41058aac68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fm4lg" Nov 02 01:23:23.399711 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:23.399694 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99cc3c61-f501-4a64-8f7e-0f41058aac68-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-fm4lg\" (UID: \"99cc3c61-f501-4a64-8f7e-0f41058aac68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fm4lg" Nov 02 01:23:23.409393 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:23.409348 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99cc3c61-f501-4a64-8f7e-0f41058aac68-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-fm4lg\" (UID: \"99cc3c61-f501-4a64-8f7e-0f41058aac68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fm4lg" Nov 02 01:23:23.409490 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:23.409396 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rw4m\" (UniqueName: \"kubernetes.io/projected/99cc3c61-f501-4a64-8f7e-0f41058aac68-kube-api-access-5rw4m\") pod \"maas-default-gateway-openshift-default-76597d6855-fm4lg\" (UID: \"99cc3c61-f501-4a64-8f7e-0f41058aac68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fm4lg" Nov 02 01:23:23.498789 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:23.498697 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fm4lg" Nov 02 01:23:23.783197 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:23.783172 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fm4lg"] Nov 02 01:23:23.785333 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:23:23.785289 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99cc3c61_f501_4a64_8f7e_0f41058aac68.slice/crio-b0bc9d45d0aa63516c236033d95d5c596e21f4b0517626dc2744b48eb77858f3 WatchSource:0}: Error finding container b0bc9d45d0aa63516c236033d95d5c596e21f4b0517626dc2744b48eb77858f3: Status 404 returned error can't find the container with id b0bc9d45d0aa63516c236033d95d5c596e21f4b0517626dc2744b48eb77858f3 Nov 02 01:23:24.665511 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:24.665472 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fm4lg" event={"ID":"99cc3c61-f501-4a64-8f7e-0f41058aac68","Type":"ContainerStarted","Data":"b0bc9d45d0aa63516c236033d95d5c596e21f4b0517626dc2744b48eb77858f3"} Nov 02 01:23:27.201412 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:27.201361 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:23:27.201675 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:27.201454 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:23:27.201675 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:27.201482 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:23:27.688468 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:27.688430 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fm4lg" event={"ID":"99cc3c61-f501-4a64-8f7e-0f41058aac68","Type":"ContainerStarted","Data":"c5e8b961667bc74f16c7e4da9d24aee04c4219311409da920d3a8a826adb3b87"} Nov 02 01:23:27.713695 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:27.713638 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fm4lg" podStartSLOduration=1.300531057 podStartE2EDuration="4.713623881s" totalImagesPullingTime="3.413092824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:23:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:23:27.711714047 +0000 UTC m=+512.332719258" watchObservedRunningTime="2026-11-02 01:23:27.713623881 +0000 UTC m=+512.334629093" Nov 02 01:23:28.499117 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:28.499087 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fm4lg" Nov 02 01:23:28.500753 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:28.500720 4427 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-fm4lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Nov 02 01:23:28.500898 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:28.500770 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fm4lg" podUID="99cc3c61-f501-4a64-8f7e-0f41058aac68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Nov 02 01:23:29.499985 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:29.499942 4427 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-fm4lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Nov 02 01:23:29.500424 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:29.500004 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fm4lg" podUID="99cc3c61-f501-4a64-8f7e-0f41058aac68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Nov 02 01:23:29.697820 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:29.697787 4427 generic.go:356] "Generic (PLEG): container finished" podID="bcf15fa4-15f1-4dc9-8755-9bcccfdc846b" containerID="c22144c1728a9529c65802199385bc79bf0d1822b08839703133f102f0153502" exitCode=1 Nov 02 01:23:29.698002 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:29.697858 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-587977945b-ktcgw" event={"ID":"bcf15fa4-15f1-4dc9-8755-9bcccfdc846b","Type":"ContainerDied","Data":"c22144c1728a9529c65802199385bc79bf0d1822b08839703133f102f0153502"} Nov 02 01:23:29.698348 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:29.698329 4427 scope.go:122] "RemoveContainer" containerID="c22144c1728a9529c65802199385bc79bf0d1822b08839703133f102f0153502" Nov 02 01:23:30.499941 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:30.499908 4427 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-fm4lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Nov 02 01:23:30.500107 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:30.499965 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fm4lg" podUID="99cc3c61-f501-4a64-8f7e-0f41058aac68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Nov 02 01:23:30.704989 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:30.704958 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-587977945b-ktcgw" event={"ID":"bcf15fa4-15f1-4dc9-8755-9bcccfdc846b","Type":"ContainerStarted","Data":"f7328556dd4f638e90722886ed40b7848bcaae2ee83c6f45a36405d8ee6166f2"} Nov 02 01:23:30.705189 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:30.705169 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-587977945b-ktcgw" Nov 02 01:23:31.499445 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:31.499409 4427 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-fm4lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Nov 02 01:23:31.499625 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:31.499460 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fm4lg" podUID="99cc3c61-f501-4a64-8f7e-0f41058aac68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Nov 02 01:23:31.606534 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:23:31.606496 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:23:32.499898 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:32.499860 4427 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-fm4lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Nov 02 01:23:32.500100 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:32.499914 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fm4lg" podUID="99cc3c61-f501-4a64-8f7e-0f41058aac68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Nov 02 01:23:33.499897 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:33.499860 4427 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-fm4lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Nov 02 01:23:33.500289 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:33.499928 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fm4lg" podUID="99cc3c61-f501-4a64-8f7e-0f41058aac68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Nov 02 01:23:34.499716 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:34.499677 4427 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-fm4lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Nov 02 01:23:34.499897 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:34.499728 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fm4lg" podUID="99cc3c61-f501-4a64-8f7e-0f41058aac68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Nov 02 01:23:35.502941 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:35.502912 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fm4lg" Nov 02 01:23:35.503355 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:35.503176 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fm4lg" Nov 02 01:23:35.503986 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:35.503961 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fm4lg" Nov 02 01:23:38.206945 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:38.206914 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7l9hx"] Nov 02 01:23:38.210500 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:38.210471 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7l9hx" Nov 02 01:23:38.225559 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:38.225523 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7l9hx"] Nov 02 01:23:38.255612 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:38.255578 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a6ab889-0f65-45fa-b105-5d5240243eb8-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-7l9hx\" (UID: \"2a6ab889-0f65-45fa-b105-5d5240243eb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7l9hx" Nov 02 01:23:38.255774 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:38.255619 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a6ab889-0f65-45fa-b105-5d5240243eb8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-7l9hx\" (UID: \"2a6ab889-0f65-45fa-b105-5d5240243eb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7l9hx" Nov 02 01:23:38.255774 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:38.255645 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a6ab889-0f65-45fa-b105-5d5240243eb8-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-7l9hx\" (UID: \"2a6ab889-0f65-45fa-b105-5d5240243eb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7l9hx" Nov 02 01:23:38.255774 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:38.255728 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a6ab889-0f65-45fa-b105-5d5240243eb8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-7l9hx\" (UID: \"2a6ab889-0f65-45fa-b105-5d5240243eb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7l9hx" Nov 02 01:23:38.255774 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:38.255759 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a6ab889-0f65-45fa-b105-5d5240243eb8-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-7l9hx\" (UID: \"2a6ab889-0f65-45fa-b105-5d5240243eb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7l9hx" Nov 02 01:23:38.255936 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:38.255785 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvwnd\" (UniqueName: \"kubernetes.io/projected/2a6ab889-0f65-45fa-b105-5d5240243eb8-kube-api-access-gvwnd\") pod \"maas-default-gateway-openshift-default-76597d6855-7l9hx\" (UID: \"2a6ab889-0f65-45fa-b105-5d5240243eb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7l9hx" Nov 02 01:23:38.255936 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:38.255920 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a6ab889-0f65-45fa-b105-5d5240243eb8-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-7l9hx\" (UID: \"2a6ab889-0f65-45fa-b105-5d5240243eb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7l9hx" Nov 02 01:23:38.256015 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:38.255960 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a6ab889-0f65-45fa-b105-5d5240243eb8-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-7l9hx\" (UID: \"2a6ab889-0f65-45fa-b105-5d5240243eb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7l9hx" Nov 02 01:23:38.256062 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:38.256028 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a6ab889-0f65-45fa-b105-5d5240243eb8-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-7l9hx\" (UID: \"2a6ab889-0f65-45fa-b105-5d5240243eb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7l9hx" Nov 02 01:23:38.356468 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:38.356428 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a6ab889-0f65-45fa-b105-5d5240243eb8-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-7l9hx\" (UID: \"2a6ab889-0f65-45fa-b105-5d5240243eb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7l9hx" Nov 02 01:23:38.356677 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:38.356493 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a6ab889-0f65-45fa-b105-5d5240243eb8-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-7l9hx\" (UID: \"2a6ab889-0f65-45fa-b105-5d5240243eb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7l9hx" Nov 02 01:23:38.356677 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:38.356527 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a6ab889-0f65-45fa-b105-5d5240243eb8-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-7l9hx\" (UID: \"2a6ab889-0f65-45fa-b105-5d5240243eb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7l9hx" Nov 02 01:23:38.356677 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:38.356550 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a6ab889-0f65-45fa-b105-5d5240243eb8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-7l9hx\" (UID: \"2a6ab889-0f65-45fa-b105-5d5240243eb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7l9hx" Nov 02 01:23:38.356677 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:38.356571 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a6ab889-0f65-45fa-b105-5d5240243eb8-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-7l9hx\" (UID: \"2a6ab889-0f65-45fa-b105-5d5240243eb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7l9hx" Nov 02 01:23:38.356903 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:38.356717 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a6ab889-0f65-45fa-b105-5d5240243eb8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-7l9hx\" (UID: \"2a6ab889-0f65-45fa-b105-5d5240243eb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7l9hx" Nov 02 01:23:38.356903 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:38.356768 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a6ab889-0f65-45fa-b105-5d5240243eb8-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-7l9hx\" (UID: \"2a6ab889-0f65-45fa-b105-5d5240243eb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7l9hx" Nov 02 01:23:38.356903 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:38.356807 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvwnd\" (UniqueName: \"kubernetes.io/projected/2a6ab889-0f65-45fa-b105-5d5240243eb8-kube-api-access-gvwnd\") pod \"maas-default-gateway-openshift-default-76597d6855-7l9hx\" (UID: \"2a6ab889-0f65-45fa-b105-5d5240243eb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7l9hx" Nov 02 01:23:38.356903 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:38.356883 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a6ab889-0f65-45fa-b105-5d5240243eb8-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-7l9hx\" (UID: \"2a6ab889-0f65-45fa-b105-5d5240243eb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7l9hx" Nov 02 01:23:38.357135 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:38.356909 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a6ab889-0f65-45fa-b105-5d5240243eb8-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-7l9hx\" (UID: \"2a6ab889-0f65-45fa-b105-5d5240243eb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7l9hx" Nov 02 01:23:38.357135 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:38.357063 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a6ab889-0f65-45fa-b105-5d5240243eb8-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-7l9hx\" (UID: \"2a6ab889-0f65-45fa-b105-5d5240243eb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7l9hx" Nov 02 01:23:38.357240 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:38.357217 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a6ab889-0f65-45fa-b105-5d5240243eb8-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-7l9hx\" (UID: \"2a6ab889-0f65-45fa-b105-5d5240243eb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7l9hx" Nov 02 01:23:38.357439 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:38.357417 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a6ab889-0f65-45fa-b105-5d5240243eb8-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-7l9hx\" (UID: \"2a6ab889-0f65-45fa-b105-5d5240243eb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7l9hx" Nov 02 01:23:38.357628 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:38.357583 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a6ab889-0f65-45fa-b105-5d5240243eb8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-7l9hx\" (UID: \"2a6ab889-0f65-45fa-b105-5d5240243eb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7l9hx" Nov 02 01:23:38.359152 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:38.359125 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a6ab889-0f65-45fa-b105-5d5240243eb8-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-7l9hx\" (UID: \"2a6ab889-0f65-45fa-b105-5d5240243eb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7l9hx" Nov 02 01:23:38.359238 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:38.359167 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a6ab889-0f65-45fa-b105-5d5240243eb8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-7l9hx\" (UID: \"2a6ab889-0f65-45fa-b105-5d5240243eb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7l9hx" Nov 02 01:23:38.369164 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:38.369135 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a6ab889-0f65-45fa-b105-5d5240243eb8-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-7l9hx\" (UID: \"2a6ab889-0f65-45fa-b105-5d5240243eb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7l9hx" Nov 02 01:23:38.369349 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:38.369329 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvwnd\" (UniqueName: \"kubernetes.io/projected/2a6ab889-0f65-45fa-b105-5d5240243eb8-kube-api-access-gvwnd\") pod \"maas-default-gateway-openshift-default-76597d6855-7l9hx\" (UID: \"2a6ab889-0f65-45fa-b105-5d5240243eb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7l9hx" Nov 02 01:23:38.523094 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:38.522994 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7l9hx" Nov 02 01:23:38.802728 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:38.802699 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7l9hx"] Nov 02 01:23:38.805102 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:23:38.805073 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a6ab889_0f65_45fa_b105_5d5240243eb8.slice/crio-6efcae6ab0e247173936c1447234a2b9ec8bc610a1baa703d342a4bf1e6cb4d9 WatchSource:0}: Error finding container 6efcae6ab0e247173936c1447234a2b9ec8bc610a1baa703d342a4bf1e6cb4d9: Status 404 returned error can't find the container with id 6efcae6ab0e247173936c1447234a2b9ec8bc610a1baa703d342a4bf1e6cb4d9 Nov 02 01:23:38.807906 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:38.807874 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:23:38.807978 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:38.807948 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:23:38.808017 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:38.807978 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:23:39.744021 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:39.743984 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7l9hx" event={"ID":"2a6ab889-0f65-45fa-b105-5d5240243eb8","Type":"ContainerStarted","Data":"b875734662eb4e72a09b5f4d58a3c64bb5b5599396a9f1d0187ce55cb4550c9b"} Nov 02 01:23:39.744440 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:39.744042 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7l9hx" event={"ID":"2a6ab889-0f65-45fa-b105-5d5240243eb8","Type":"ContainerStarted","Data":"6efcae6ab0e247173936c1447234a2b9ec8bc610a1baa703d342a4bf1e6cb4d9"} Nov 02 01:23:39.780400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:39.779695 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7l9hx" podStartSLOduration=1.779671992 podStartE2EDuration="1.779671992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:23:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:23:39.773759917 +0000 UTC m=+524.394765140" watchObservedRunningTime="2026-11-02 01:23:39.779671992 +0000 UTC m=+524.400677204" Nov 02 01:23:40.523976 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:40.523938 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7l9hx" Nov 02 01:23:40.525506 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:40.525471 4427 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-7l9hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Nov 02 01:23:40.525649 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:40.525519 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7l9hx" podUID="2a6ab889-0f65-45fa-b105-5d5240243eb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Nov 02 01:23:41.524413 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:41.524348 4427 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-7l9hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Nov 02 01:23:41.524930 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:41.524453 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7l9hx" podUID="2a6ab889-0f65-45fa-b105-5d5240243eb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Nov 02 01:23:41.711567 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:41.711538 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-587977945b-ktcgw" Nov 02 01:23:42.524474 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:42.524429 4427 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-7l9hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Nov 02 01:23:42.524978 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:42.524494 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7l9hx" podUID="2a6ab889-0f65-45fa-b105-5d5240243eb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Nov 02 01:23:43.524241 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:43.524203 4427 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-7l9hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Nov 02 01:23:43.524458 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:43.524261 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7l9hx" podUID="2a6ab889-0f65-45fa-b105-5d5240243eb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Nov 02 01:23:44.523904 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:44.523869 4427 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-7l9hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Nov 02 01:23:44.524338 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:44.523931 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7l9hx" podUID="2a6ab889-0f65-45fa-b105-5d5240243eb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Nov 02 01:23:45.524399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:45.524330 4427 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-7l9hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Nov 02 01:23:45.524797 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:45.524436 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7l9hx" podUID="2a6ab889-0f65-45fa-b105-5d5240243eb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Nov 02 01:23:47.043417 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:47.043329 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7l9hx" Nov 02 01:23:47.043904 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:47.043745 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7l9hx" Nov 02 01:23:47.044547 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:47.044529 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7l9hx" Nov 02 01:23:53.221592 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:53.221548 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-c8lsb"] Nov 02 01:23:53.228127 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:53.228095 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-c8lsb" Nov 02 01:23:53.243746 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:53.243699 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-c8lsb"] Nov 02 01:23:53.322968 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:53.322894 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d77ae499-892d-442f-83de-007ef22dfdcc-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-c8lsb\" (UID: \"d77ae499-892d-442f-83de-007ef22dfdcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-c8lsb" Nov 02 01:23:53.322968 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:53.322971 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d77ae499-892d-442f-83de-007ef22dfdcc-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-c8lsb\" (UID: \"d77ae499-892d-442f-83de-007ef22dfdcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-c8lsb" Nov 02 01:23:53.323270 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:53.323085 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d77ae499-892d-442f-83de-007ef22dfdcc-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-c8lsb\" (UID: \"d77ae499-892d-442f-83de-007ef22dfdcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-c8lsb" Nov 02 01:23:53.323270 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:53.323145 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d77ae499-892d-442f-83de-007ef22dfdcc-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-c8lsb\" (UID: \"d77ae499-892d-442f-83de-007ef22dfdcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-c8lsb" Nov 02 01:23:53.323270 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:53.323227 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d77ae499-892d-442f-83de-007ef22dfdcc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-c8lsb\" (UID: \"d77ae499-892d-442f-83de-007ef22dfdcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-c8lsb" Nov 02 01:23:53.323491 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:53.323292 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d77ae499-892d-442f-83de-007ef22dfdcc-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-c8lsb\" (UID: \"d77ae499-892d-442f-83de-007ef22dfdcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-c8lsb" Nov 02 01:23:53.323491 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:53.323436 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfzqc\" (UniqueName: \"kubernetes.io/projected/d77ae499-892d-442f-83de-007ef22dfdcc-kube-api-access-jfzqc\") pod \"maas-default-gateway-openshift-default-76597d6855-c8lsb\" (UID: \"d77ae499-892d-442f-83de-007ef22dfdcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-c8lsb" Nov 02 01:23:53.323736 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:53.323487 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d77ae499-892d-442f-83de-007ef22dfdcc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-c8lsb\" (UID: \"d77ae499-892d-442f-83de-007ef22dfdcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-c8lsb" Nov 02 01:23:53.323736 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:53.323602 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d77ae499-892d-442f-83de-007ef22dfdcc-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-c8lsb\" (UID: \"d77ae499-892d-442f-83de-007ef22dfdcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-c8lsb" Nov 02 01:23:53.424733 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:53.424681 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d77ae499-892d-442f-83de-007ef22dfdcc-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-c8lsb\" (UID: \"d77ae499-892d-442f-83de-007ef22dfdcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-c8lsb" Nov 02 01:23:53.424733 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:53.424745 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfzqc\" (UniqueName: \"kubernetes.io/projected/d77ae499-892d-442f-83de-007ef22dfdcc-kube-api-access-jfzqc\") pod \"maas-default-gateway-openshift-default-76597d6855-c8lsb\" (UID: \"d77ae499-892d-442f-83de-007ef22dfdcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-c8lsb" Nov 02 01:23:53.424999 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:53.424773 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d77ae499-892d-442f-83de-007ef22dfdcc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-c8lsb\" (UID: \"d77ae499-892d-442f-83de-007ef22dfdcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-c8lsb" Nov 02 01:23:53.424999 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:53.424797 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d77ae499-892d-442f-83de-007ef22dfdcc-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-c8lsb\" (UID: \"d77ae499-892d-442f-83de-007ef22dfdcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-c8lsb" Nov 02 01:23:53.424999 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:53.424837 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d77ae499-892d-442f-83de-007ef22dfdcc-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-c8lsb\" (UID: \"d77ae499-892d-442f-83de-007ef22dfdcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-c8lsb" Nov 02 01:23:53.424999 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:53.424850 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d77ae499-892d-442f-83de-007ef22dfdcc-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-c8lsb\" (UID: \"d77ae499-892d-442f-83de-007ef22dfdcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-c8lsb" Nov 02 01:23:53.424999 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:53.424872 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d77ae499-892d-442f-83de-007ef22dfdcc-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-c8lsb\" (UID: \"d77ae499-892d-442f-83de-007ef22dfdcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-c8lsb" Nov 02 01:23:53.424999 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:53.424897 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d77ae499-892d-442f-83de-007ef22dfdcc-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-c8lsb\" (UID: \"d77ae499-892d-442f-83de-007ef22dfdcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-c8lsb" Nov 02 01:23:53.424999 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:53.424921 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d77ae499-892d-442f-83de-007ef22dfdcc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-c8lsb\" (UID: \"d77ae499-892d-442f-83de-007ef22dfdcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-c8lsb" Nov 02 01:23:53.425350 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:53.425250 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d77ae499-892d-442f-83de-007ef22dfdcc-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-c8lsb\" (UID: \"d77ae499-892d-442f-83de-007ef22dfdcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-c8lsb" Nov 02 01:23:53.425350 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:53.425301 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d77ae499-892d-442f-83de-007ef22dfdcc-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-c8lsb\" (UID: \"d77ae499-892d-442f-83de-007ef22dfdcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-c8lsb" Nov 02 01:23:53.425506 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:53.425426 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d77ae499-892d-442f-83de-007ef22dfdcc-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-c8lsb\" (UID: \"d77ae499-892d-442f-83de-007ef22dfdcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-c8lsb" Nov 02 01:23:53.425506 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:53.425480 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d77ae499-892d-442f-83de-007ef22dfdcc-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-c8lsb\" (UID: \"d77ae499-892d-442f-83de-007ef22dfdcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-c8lsb" Nov 02 01:23:53.425627 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:53.425608 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d77ae499-892d-442f-83de-007ef22dfdcc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-c8lsb\" (UID: \"d77ae499-892d-442f-83de-007ef22dfdcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-c8lsb" Nov 02 01:23:53.427457 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:53.427434 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d77ae499-892d-442f-83de-007ef22dfdcc-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-c8lsb\" (UID: \"d77ae499-892d-442f-83de-007ef22dfdcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-c8lsb" Nov 02 01:23:53.427707 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:53.427688 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d77ae499-892d-442f-83de-007ef22dfdcc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-c8lsb\" (UID: \"d77ae499-892d-442f-83de-007ef22dfdcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-c8lsb" Nov 02 01:23:53.437133 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:53.437103 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d77ae499-892d-442f-83de-007ef22dfdcc-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-c8lsb\" (UID: \"d77ae499-892d-442f-83de-007ef22dfdcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-c8lsb" Nov 02 01:23:53.437307 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:53.437290 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfzqc\" (UniqueName: \"kubernetes.io/projected/d77ae499-892d-442f-83de-007ef22dfdcc-kube-api-access-jfzqc\") pod \"maas-default-gateway-openshift-default-76597d6855-c8lsb\" (UID: \"d77ae499-892d-442f-83de-007ef22dfdcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-c8lsb" Nov 02 01:23:53.555130 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:53.555030 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-c8lsb" Nov 02 01:23:53.836744 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:53.836721 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-c8lsb"] Nov 02 01:23:53.838556 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:23:53.838522 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd77ae499_892d_442f_83de_007ef22dfdcc.slice/crio-2bc75f231e5a666916a21f29fb8b454e4280343bb60d27a8aa7431cfc5d21ab8 WatchSource:0}: Error finding container 2bc75f231e5a666916a21f29fb8b454e4280343bb60d27a8aa7431cfc5d21ab8: Status 404 returned error can't find the container with id 2bc75f231e5a666916a21f29fb8b454e4280343bb60d27a8aa7431cfc5d21ab8 Nov 02 01:23:53.841519 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:53.841490 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:23:53.841593 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:53.841557 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:23:53.841593 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:53.841587 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:23:54.826164 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:54.826127 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-c8lsb" event={"ID":"d77ae499-892d-442f-83de-007ef22dfdcc","Type":"ContainerStarted","Data":"84771b762c3e4b3b00bba726e885768283426f62de0894518f73f3312e5f5033"} Nov 02 01:23:54.826643 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:54.826182 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-c8lsb" event={"ID":"d77ae499-892d-442f-83de-007ef22dfdcc","Type":"ContainerStarted","Data":"2bc75f231e5a666916a21f29fb8b454e4280343bb60d27a8aa7431cfc5d21ab8"} Nov 02 01:23:54.852140 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:54.852091 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-c8lsb" podStartSLOduration=1.852078014 podStartE2EDuration="1.852078014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:23:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:23:54.850265199 +0000 UTC m=+539.471270422" watchObservedRunningTime="2026-11-02 01:23:54.852078014 +0000 UTC m=+539.473083224" Nov 02 01:23:55.556075 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:55.556033 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-c8lsb" Nov 02 01:23:55.557548 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:55.557521 4427 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-c8lsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Nov 02 01:23:55.557658 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:55.557598 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-c8lsb" podUID="d77ae499-892d-442f-83de-007ef22dfdcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Nov 02 01:23:56.555562 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:56.555520 4427 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-c8lsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Nov 02 01:23:56.555977 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:56.555583 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-c8lsb" podUID="d77ae499-892d-442f-83de-007ef22dfdcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Nov 02 01:23:57.555515 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:57.555482 4427 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-c8lsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Nov 02 01:23:57.555693 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:57.555542 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-c8lsb" podUID="d77ae499-892d-442f-83de-007ef22dfdcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Nov 02 01:23:58.556113 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:58.556077 4427 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-c8lsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Nov 02 01:23:58.556523 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:58.556130 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-c8lsb" podUID="d77ae499-892d-442f-83de-007ef22dfdcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Nov 02 01:23:59.555607 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:59.555565 4427 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-c8lsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Nov 02 01:23:59.555792 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:23:59.555625 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-c8lsb" podUID="d77ae499-892d-442f-83de-007ef22dfdcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Nov 02 01:24:00.555909 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:24:00.555863 4427 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-c8lsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Nov 02 01:24:00.556315 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:24:00.555933 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-c8lsb" podUID="d77ae499-892d-442f-83de-007ef22dfdcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Nov 02 01:24:01.582535 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:24:01.582493 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:24:02.555833 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:24:02.555788 4427 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-c8lsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 01:24:02.556012 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:24:02.555859 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-c8lsb" podUID="d77ae499-892d-442f-83de-007ef22dfdcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:24:03.597622 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:24:03.597575 4427 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-c8lsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 01:24:03.598049 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:24:03.597649 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-c8lsb" podUID="d77ae499-892d-442f-83de-007ef22dfdcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:24:04.252711 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:24:04.252676 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-c8lsb" Nov 02 01:24:04.253089 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:24:04.253065 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-c8lsb" Nov 02 01:24:04.254231 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:24:04.254213 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-c8lsb" Nov 02 01:24:31.580843 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:24:31.580801 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:25:01.582264 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:25:01.582225 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:25:08.727775 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:08.727737 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl"] Nov 02 01:25:08.732456 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:08.732428 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl" Nov 02 01:25:08.748304 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:08.748271 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl"] Nov 02 01:25:08.763092 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:08.763051 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-47dd19-openshift-default-dockercfg-tx9jn\"" Nov 02 01:25:08.834595 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:08.834554 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/282522e6-7d16-4316-bf07-5e2706155435-istiod-ca-cert\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl\" (UID: \"282522e6-7d16-4316-bf07-5e2706155435\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl" Nov 02 01:25:08.834595 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:08.834592 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/282522e6-7d16-4316-bf07-5e2706155435-istio-podinfo\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl\" (UID: \"282522e6-7d16-4316-bf07-5e2706155435\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl" Nov 02 01:25:08.834850 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:08.834703 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/282522e6-7d16-4316-bf07-5e2706155435-credential-socket\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl\" (UID: \"282522e6-7d16-4316-bf07-5e2706155435\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl" Nov 02 01:25:08.834850 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:08.834730 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/282522e6-7d16-4316-bf07-5e2706155435-istio-envoy\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl\" (UID: \"282522e6-7d16-4316-bf07-5e2706155435\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl" Nov 02 01:25:08.834850 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:08.834753 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdq8p\" (UniqueName: \"kubernetes.io/projected/282522e6-7d16-4316-bf07-5e2706155435-kube-api-access-rdq8p\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl\" (UID: \"282522e6-7d16-4316-bf07-5e2706155435\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl" Nov 02 01:25:08.834850 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:08.834822 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/282522e6-7d16-4316-bf07-5e2706155435-istio-data\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl\" (UID: \"282522e6-7d16-4316-bf07-5e2706155435\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl" Nov 02 01:25:08.835061 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:08.834897 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/282522e6-7d16-4316-bf07-5e2706155435-workload-socket\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl\" (UID: \"282522e6-7d16-4316-bf07-5e2706155435\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl" Nov 02 01:25:08.835061 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:08.834930 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/282522e6-7d16-4316-bf07-5e2706155435-workload-certs\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl\" (UID: \"282522e6-7d16-4316-bf07-5e2706155435\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl" Nov 02 01:25:08.835061 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:08.834962 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/282522e6-7d16-4316-bf07-5e2706155435-istio-token\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl\" (UID: \"282522e6-7d16-4316-bf07-5e2706155435\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl" Nov 02 01:25:08.909469 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:08.909430 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf"] Nov 02 01:25:08.913687 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:08.913665 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf" Nov 02 01:25:08.932092 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:08.932065 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf"] Nov 02 01:25:08.936429 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:08.936398 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/282522e6-7d16-4316-bf07-5e2706155435-workload-certs\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl\" (UID: \"282522e6-7d16-4316-bf07-5e2706155435\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl" Nov 02 01:25:08.936614 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:08.936442 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/282522e6-7d16-4316-bf07-5e2706155435-istio-token\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl\" (UID: \"282522e6-7d16-4316-bf07-5e2706155435\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl" Nov 02 01:25:08.936614 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:08.936513 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/282522e6-7d16-4316-bf07-5e2706155435-istiod-ca-cert\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl\" (UID: \"282522e6-7d16-4316-bf07-5e2706155435\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl" Nov 02 01:25:08.936614 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:08.936540 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/282522e6-7d16-4316-bf07-5e2706155435-istio-podinfo\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl\" (UID: \"282522e6-7d16-4316-bf07-5e2706155435\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl" Nov 02 01:25:08.936614 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:08.936600 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/282522e6-7d16-4316-bf07-5e2706155435-credential-socket\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl\" (UID: \"282522e6-7d16-4316-bf07-5e2706155435\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl" Nov 02 01:25:08.936765 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:08.936627 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/282522e6-7d16-4316-bf07-5e2706155435-istio-envoy\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl\" (UID: \"282522e6-7d16-4316-bf07-5e2706155435\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl" Nov 02 01:25:08.936765 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:08.936659 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdq8p\" (UniqueName: \"kubernetes.io/projected/282522e6-7d16-4316-bf07-5e2706155435-kube-api-access-rdq8p\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl\" (UID: \"282522e6-7d16-4316-bf07-5e2706155435\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl" Nov 02 01:25:08.936765 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:08.936691 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/282522e6-7d16-4316-bf07-5e2706155435-istio-data\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl\" (UID: \"282522e6-7d16-4316-bf07-5e2706155435\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl" Nov 02 01:25:08.936883 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:08.936768 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/282522e6-7d16-4316-bf07-5e2706155435-workload-socket\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl\" (UID: \"282522e6-7d16-4316-bf07-5e2706155435\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl" Nov 02 01:25:08.936883 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:08.936782 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/282522e6-7d16-4316-bf07-5e2706155435-workload-certs\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl\" (UID: \"282522e6-7d16-4316-bf07-5e2706155435\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl" Nov 02 01:25:08.937097 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:08.937073 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/282522e6-7d16-4316-bf07-5e2706155435-credential-socket\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl\" (UID: \"282522e6-7d16-4316-bf07-5e2706155435\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl" Nov 02 01:25:08.937321 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:08.937296 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/282522e6-7d16-4316-bf07-5e2706155435-workload-socket\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl\" (UID: \"282522e6-7d16-4316-bf07-5e2706155435\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl" Nov 02 01:25:08.938306 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:08.938280 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/282522e6-7d16-4316-bf07-5e2706155435-istio-data\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl\" (UID: \"282522e6-7d16-4316-bf07-5e2706155435\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl" Nov 02 01:25:08.938461 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:08.938438 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/282522e6-7d16-4316-bf07-5e2706155435-istiod-ca-cert\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl\" (UID: \"282522e6-7d16-4316-bf07-5e2706155435\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl" Nov 02 01:25:08.940100 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:08.940054 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/282522e6-7d16-4316-bf07-5e2706155435-istio-envoy\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl\" (UID: \"282522e6-7d16-4316-bf07-5e2706155435\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl" Nov 02 01:25:08.940779 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:08.940759 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/282522e6-7d16-4316-bf07-5e2706155435-istio-podinfo\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl\" (UID: \"282522e6-7d16-4316-bf07-5e2706155435\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl" Nov 02 01:25:08.948787 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:08.948748 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-123986-openshift-default-dockercfg-8gxrz\"" Nov 02 01:25:08.955732 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:08.955703 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/282522e6-7d16-4316-bf07-5e2706155435-istio-token\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl\" (UID: \"282522e6-7d16-4316-bf07-5e2706155435\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl" Nov 02 01:25:08.955884 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:08.955806 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdq8p\" (UniqueName: \"kubernetes.io/projected/282522e6-7d16-4316-bf07-5e2706155435-kube-api-access-rdq8p\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl\" (UID: \"282522e6-7d16-4316-bf07-5e2706155435\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl" Nov 02 01:25:09.038211 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.038126 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b880c081-4957-4151-9e1f-fb8f397dedb3-istio-token\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf\" (UID: \"b880c081-4957-4151-9e1f-fb8f397dedb3\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf" Nov 02 01:25:09.038211 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.038196 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b880c081-4957-4151-9e1f-fb8f397dedb3-workload-certs\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf\" (UID: \"b880c081-4957-4151-9e1f-fb8f397dedb3\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf" Nov 02 01:25:09.038526 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.038287 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b880c081-4957-4151-9e1f-fb8f397dedb3-workload-socket\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf\" (UID: \"b880c081-4957-4151-9e1f-fb8f397dedb3\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf" Nov 02 01:25:09.038526 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.038321 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b880c081-4957-4151-9e1f-fb8f397dedb3-istiod-ca-cert\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf\" (UID: \"b880c081-4957-4151-9e1f-fb8f397dedb3\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf" Nov 02 01:25:09.038526 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.038404 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5nq9\" (UniqueName: \"kubernetes.io/projected/b880c081-4957-4151-9e1f-fb8f397dedb3-kube-api-access-l5nq9\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf\" (UID: \"b880c081-4957-4151-9e1f-fb8f397dedb3\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf" Nov 02 01:25:09.038526 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.038499 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b880c081-4957-4151-9e1f-fb8f397dedb3-istio-data\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf\" (UID: \"b880c081-4957-4151-9e1f-fb8f397dedb3\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf" Nov 02 01:25:09.038753 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.038643 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b880c081-4957-4151-9e1f-fb8f397dedb3-istio-envoy\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf\" (UID: \"b880c081-4957-4151-9e1f-fb8f397dedb3\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf" Nov 02 01:25:09.038753 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.038678 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b880c081-4957-4151-9e1f-fb8f397dedb3-istio-podinfo\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf\" (UID: \"b880c081-4957-4151-9e1f-fb8f397dedb3\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf" Nov 02 01:25:09.038860 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.038763 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b880c081-4957-4151-9e1f-fb8f397dedb3-credential-socket\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf\" (UID: \"b880c081-4957-4151-9e1f-fb8f397dedb3\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf" Nov 02 01:25:09.049084 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.049056 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl" Nov 02 01:25:09.082471 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.082429 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9"] Nov 02 01:25:09.091005 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.090977 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9" Nov 02 01:25:09.101401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.100621 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9"] Nov 02 01:25:09.105523 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.105280 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-ca1c53-openshift-default-dockercfg-hhbph\"" Nov 02 01:25:09.140158 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.140122 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19218597-2356-4f9d-bf2b-ac4597da575a-credential-socket\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9\" (UID: \"19218597-2356-4f9d-bf2b-ac4597da575a\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9" Nov 02 01:25:09.140306 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.140171 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19218597-2356-4f9d-bf2b-ac4597da575a-workload-certs\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9\" (UID: \"19218597-2356-4f9d-bf2b-ac4597da575a\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9" Nov 02 01:25:09.140306 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.140199 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjgpt\" (UniqueName: \"kubernetes.io/projected/19218597-2356-4f9d-bf2b-ac4597da575a-kube-api-access-wjgpt\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9\" (UID: \"19218597-2356-4f9d-bf2b-ac4597da575a\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9" Nov 02 01:25:09.140306 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.140249 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b880c081-4957-4151-9e1f-fb8f397dedb3-workload-certs\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf\" (UID: \"b880c081-4957-4151-9e1f-fb8f397dedb3\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf" Nov 02 01:25:09.140306 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.140280 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19218597-2356-4f9d-bf2b-ac4597da575a-workload-socket\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9\" (UID: \"19218597-2356-4f9d-bf2b-ac4597da575a\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9" Nov 02 01:25:09.140456 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.140319 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b880c081-4957-4151-9e1f-fb8f397dedb3-workload-socket\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf\" (UID: \"b880c081-4957-4151-9e1f-fb8f397dedb3\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf" Nov 02 01:25:09.140456 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.140346 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b880c081-4957-4151-9e1f-fb8f397dedb3-istiod-ca-cert\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf\" (UID: \"b880c081-4957-4151-9e1f-fb8f397dedb3\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf" Nov 02 01:25:09.140456 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.140420 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5nq9\" (UniqueName: \"kubernetes.io/projected/b880c081-4957-4151-9e1f-fb8f397dedb3-kube-api-access-l5nq9\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf\" (UID: \"b880c081-4957-4151-9e1f-fb8f397dedb3\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf" Nov 02 01:25:09.140542 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.140473 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b880c081-4957-4151-9e1f-fb8f397dedb3-istio-data\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf\" (UID: \"b880c081-4957-4151-9e1f-fb8f397dedb3\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf" Nov 02 01:25:09.140542 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.140516 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b880c081-4957-4151-9e1f-fb8f397dedb3-istio-envoy\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf\" (UID: \"b880c081-4957-4151-9e1f-fb8f397dedb3\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf" Nov 02 01:25:09.140600 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.140542 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b880c081-4957-4151-9e1f-fb8f397dedb3-istio-podinfo\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf\" (UID: \"b880c081-4957-4151-9e1f-fb8f397dedb3\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf" Nov 02 01:25:09.140632 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.140609 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19218597-2356-4f9d-bf2b-ac4597da575a-istiod-ca-cert\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9\" (UID: \"19218597-2356-4f9d-bf2b-ac4597da575a\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9" Nov 02 01:25:09.140664 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.140650 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19218597-2356-4f9d-bf2b-ac4597da575a-istio-token\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9\" (UID: \"19218597-2356-4f9d-bf2b-ac4597da575a\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9" Nov 02 01:25:09.140742 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.140684 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b880c081-4957-4151-9e1f-fb8f397dedb3-credential-socket\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf\" (UID: \"b880c081-4957-4151-9e1f-fb8f397dedb3\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf" Nov 02 01:25:09.140742 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.140712 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19218597-2356-4f9d-bf2b-ac4597da575a-istio-data\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9\" (UID: \"19218597-2356-4f9d-bf2b-ac4597da575a\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9" Nov 02 01:25:09.140804 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.140743 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19218597-2356-4f9d-bf2b-ac4597da575a-istio-envoy\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9\" (UID: \"19218597-2356-4f9d-bf2b-ac4597da575a\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9" Nov 02 01:25:09.140804 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.140770 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b880c081-4957-4151-9e1f-fb8f397dedb3-istio-token\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf\" (UID: \"b880c081-4957-4151-9e1f-fb8f397dedb3\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf" Nov 02 01:25:09.140804 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.140795 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19218597-2356-4f9d-bf2b-ac4597da575a-istio-podinfo\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9\" (UID: \"19218597-2356-4f9d-bf2b-ac4597da575a\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9" Nov 02 01:25:09.141028 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.141005 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b880c081-4957-4151-9e1f-fb8f397dedb3-workload-socket\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf\" (UID: \"b880c081-4957-4151-9e1f-fb8f397dedb3\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf" Nov 02 01:25:09.141352 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.141333 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b880c081-4957-4151-9e1f-fb8f397dedb3-workload-certs\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf\" (UID: \"b880c081-4957-4151-9e1f-fb8f397dedb3\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf" Nov 02 01:25:09.141614 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.141584 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b880c081-4957-4151-9e1f-fb8f397dedb3-istiod-ca-cert\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf\" (UID: \"b880c081-4957-4151-9e1f-fb8f397dedb3\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf" Nov 02 01:25:09.141903 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.141880 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b880c081-4957-4151-9e1f-fb8f397dedb3-credential-socket\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf\" (UID: \"b880c081-4957-4151-9e1f-fb8f397dedb3\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf" Nov 02 01:25:09.141994 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.141956 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b880c081-4957-4151-9e1f-fb8f397dedb3-istio-data\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf\" (UID: \"b880c081-4957-4151-9e1f-fb8f397dedb3\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf" Nov 02 01:25:09.145424 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.144427 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b880c081-4957-4151-9e1f-fb8f397dedb3-istio-envoy\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf\" (UID: \"b880c081-4957-4151-9e1f-fb8f397dedb3\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf" Nov 02 01:25:09.145424 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.144757 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b880c081-4957-4151-9e1f-fb8f397dedb3-istio-podinfo\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf\" (UID: \"b880c081-4957-4151-9e1f-fb8f397dedb3\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf" Nov 02 01:25:09.154623 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.154597 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b880c081-4957-4151-9e1f-fb8f397dedb3-istio-token\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf\" (UID: \"b880c081-4957-4151-9e1f-fb8f397dedb3\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf" Nov 02 01:25:09.154961 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.154935 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5nq9\" (UniqueName: \"kubernetes.io/projected/b880c081-4957-4151-9e1f-fb8f397dedb3-kube-api-access-l5nq9\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf\" (UID: \"b880c081-4957-4151-9e1f-fb8f397dedb3\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf" Nov 02 01:25:09.226533 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.226496 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf" Nov 02 01:25:09.242439 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.242392 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19218597-2356-4f9d-bf2b-ac4597da575a-istiod-ca-cert\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9\" (UID: \"19218597-2356-4f9d-bf2b-ac4597da575a\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9" Nov 02 01:25:09.242557 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.242447 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19218597-2356-4f9d-bf2b-ac4597da575a-istio-token\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9\" (UID: \"19218597-2356-4f9d-bf2b-ac4597da575a\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9" Nov 02 01:25:09.242557 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.242485 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19218597-2356-4f9d-bf2b-ac4597da575a-istio-data\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9\" (UID: \"19218597-2356-4f9d-bf2b-ac4597da575a\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9" Nov 02 01:25:09.242557 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.242511 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19218597-2356-4f9d-bf2b-ac4597da575a-istio-envoy\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9\" (UID: \"19218597-2356-4f9d-bf2b-ac4597da575a\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9" Nov 02 01:25:09.242557 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.242541 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19218597-2356-4f9d-bf2b-ac4597da575a-istio-podinfo\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9\" (UID: \"19218597-2356-4f9d-bf2b-ac4597da575a\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9" Nov 02 01:25:09.242778 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.242602 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19218597-2356-4f9d-bf2b-ac4597da575a-credential-socket\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9\" (UID: \"19218597-2356-4f9d-bf2b-ac4597da575a\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9" Nov 02 01:25:09.242778 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.242625 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19218597-2356-4f9d-bf2b-ac4597da575a-workload-certs\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9\" (UID: \"19218597-2356-4f9d-bf2b-ac4597da575a\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9" Nov 02 01:25:09.242778 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.242653 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjgpt\" (UniqueName: \"kubernetes.io/projected/19218597-2356-4f9d-bf2b-ac4597da575a-kube-api-access-wjgpt\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9\" (UID: \"19218597-2356-4f9d-bf2b-ac4597da575a\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9" Nov 02 01:25:09.242778 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.242709 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19218597-2356-4f9d-bf2b-ac4597da575a-workload-socket\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9\" (UID: \"19218597-2356-4f9d-bf2b-ac4597da575a\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9" Nov 02 01:25:09.243803 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.243180 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19218597-2356-4f9d-bf2b-ac4597da575a-workload-socket\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9\" (UID: \"19218597-2356-4f9d-bf2b-ac4597da575a\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9" Nov 02 01:25:09.243803 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.243345 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19218597-2356-4f9d-bf2b-ac4597da575a-credential-socket\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9\" (UID: \"19218597-2356-4f9d-bf2b-ac4597da575a\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9" Nov 02 01:25:09.243803 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.243493 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19218597-2356-4f9d-bf2b-ac4597da575a-workload-certs\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9\" (UID: \"19218597-2356-4f9d-bf2b-ac4597da575a\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9" Nov 02 01:25:09.243803 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.243760 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19218597-2356-4f9d-bf2b-ac4597da575a-istio-data\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9\" (UID: \"19218597-2356-4f9d-bf2b-ac4597da575a\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9" Nov 02 01:25:09.244134 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.244091 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19218597-2356-4f9d-bf2b-ac4597da575a-istiod-ca-cert\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9\" (UID: \"19218597-2356-4f9d-bf2b-ac4597da575a\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9" Nov 02 01:25:09.245953 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.245927 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19218597-2356-4f9d-bf2b-ac4597da575a-istio-podinfo\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9\" (UID: \"19218597-2356-4f9d-bf2b-ac4597da575a\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9" Nov 02 01:25:09.246500 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.246481 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19218597-2356-4f9d-bf2b-ac4597da575a-istio-envoy\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9\" (UID: \"19218597-2356-4f9d-bf2b-ac4597da575a\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9" Nov 02 01:25:09.257301 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.257275 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19218597-2356-4f9d-bf2b-ac4597da575a-istio-token\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9\" (UID: \"19218597-2356-4f9d-bf2b-ac4597da575a\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9" Nov 02 01:25:09.257770 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.257752 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjgpt\" (UniqueName: \"kubernetes.io/projected/19218597-2356-4f9d-bf2b-ac4597da575a-kube-api-access-wjgpt\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9\" (UID: \"19218597-2356-4f9d-bf2b-ac4597da575a\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9" Nov 02 01:25:09.349008 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.348911 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl"] Nov 02 01:25:09.357280 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:25:09.357244 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod282522e6_7d16_4316_bf07_5e2706155435.slice/crio-b14d07df9a7e5f68cb94b3fd1fcbece2e9ad3fb58c196818fa2bb5f3bc8d04b4 WatchSource:0}: Error finding container b14d07df9a7e5f68cb94b3fd1fcbece2e9ad3fb58c196818fa2bb5f3bc8d04b4: Status 404 returned error can't find the container with id b14d07df9a7e5f68cb94b3fd1fcbece2e9ad3fb58c196818fa2bb5f3bc8d04b4 Nov 02 01:25:09.360741 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.360697 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:25:09.360853 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.360793 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:25:09.360853 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.360832 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:25:09.420349 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.420310 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9" Nov 02 01:25:09.527954 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.527683 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf"] Nov 02 01:25:09.531132 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:25:09.531035 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb880c081_4957_4151_9e1f_fb8f397dedb3.slice/crio-bb8535ae02beb1f941e50178793aa4418bef0b5be7435322579b67172a1cb123 WatchSource:0}: Error finding container bb8535ae02beb1f941e50178793aa4418bef0b5be7435322579b67172a1cb123: Status 404 returned error can't find the container with id bb8535ae02beb1f941e50178793aa4418bef0b5be7435322579b67172a1cb123 Nov 02 01:25:09.534478 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.533974 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:25:09.534478 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.534055 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:25:09.534478 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.534101 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:25:09.709502 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.709422 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9"] Nov 02 01:25:09.711970 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:25:09.711940 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19218597_2356_4f9d_bf2b_ac4597da575a.slice/crio-bd5f9ba8d5f10d8fe6de66f4d5162454b02f0614b743422ac17970402e08fa66 WatchSource:0}: Error finding container bd5f9ba8d5f10d8fe6de66f4d5162454b02f0614b743422ac17970402e08fa66: Status 404 returned error can't find the container with id bd5f9ba8d5f10d8fe6de66f4d5162454b02f0614b743422ac17970402e08fa66 Nov 02 01:25:09.714480 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.714447 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:25:09.714555 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.714521 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:25:09.714555 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:09.714550 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:25:10.162413 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:10.158543 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl" event={"ID":"282522e6-7d16-4316-bf07-5e2706155435","Type":"ContainerStarted","Data":"3e4e3e4518a25da89666056534b7618c728efc423e571f4d498aa3205516d6ef"} Nov 02 01:25:10.162413 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:10.158644 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl" event={"ID":"282522e6-7d16-4316-bf07-5e2706155435","Type":"ContainerStarted","Data":"b14d07df9a7e5f68cb94b3fd1fcbece2e9ad3fb58c196818fa2bb5f3bc8d04b4"} Nov 02 01:25:10.162413 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:10.160274 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9" event={"ID":"19218597-2356-4f9d-bf2b-ac4597da575a","Type":"ContainerStarted","Data":"6d73244c1f5558e36ec4b736e792c4ed38240123737cf0dd0ccd5bf8a0089be7"} Nov 02 01:25:10.162413 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:10.160319 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9" event={"ID":"19218597-2356-4f9d-bf2b-ac4597da575a","Type":"ContainerStarted","Data":"bd5f9ba8d5f10d8fe6de66f4d5162454b02f0614b743422ac17970402e08fa66"} Nov 02 01:25:10.162413 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:10.161819 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf" event={"ID":"b880c081-4957-4151-9e1f-fb8f397dedb3","Type":"ContainerStarted","Data":"b5107dd9a7488440da3a86139d85e7887cda8bd9a4f18ed1e23170d1d29f81d0"} Nov 02 01:25:10.162413 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:10.161852 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf" event={"ID":"b880c081-4957-4151-9e1f-fb8f397dedb3","Type":"ContainerStarted","Data":"bb8535ae02beb1f941e50178793aa4418bef0b5be7435322579b67172a1cb123"} Nov 02 01:25:10.181610 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:10.181562 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl" podStartSLOduration=2.181548623 podStartE2EDuration="2.181548623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:25:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:25:10.180620399 +0000 UTC m=+614.801625611" watchObservedRunningTime="2026-11-02 01:25:10.181548623 +0000 UTC m=+614.802553834" Nov 02 01:25:10.203218 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:10.202969 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf" podStartSLOduration=2.202951694 podStartE2EDuration="2.202951694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:25:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:25:10.202779607 +0000 UTC m=+614.823784819" watchObservedRunningTime="2026-11-02 01:25:10.202951694 +0000 UTC m=+614.823956907" Nov 02 01:25:10.224864 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:10.224701 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9" podStartSLOduration=1.224685493 podStartE2EDuration="1.224685493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:25:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:25:10.224390308 +0000 UTC m=+614.845395513" watchObservedRunningTime="2026-11-02 01:25:10.224685493 +0000 UTC m=+614.845690705" Nov 02 01:25:10.227043 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:10.227013 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf" Nov 02 01:25:10.232152 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:10.232127 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf" Nov 02 01:25:10.421281 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:10.421188 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9" Nov 02 01:25:10.427711 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:10.427681 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9" Nov 02 01:25:11.049981 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:11.049946 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl" Nov 02 01:25:11.055577 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:11.055549 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl" Nov 02 01:25:11.167265 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:11.167145 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9" Nov 02 01:25:11.167265 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:11.167203 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl" Nov 02 01:25:11.167265 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:11.167227 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf" Nov 02 01:25:11.167996 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:11.167974 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf" Nov 02 01:25:11.168092 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:11.168033 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl" Nov 02 01:25:11.168839 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:11.168820 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9" Nov 02 01:25:16.619792 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:16.619758 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-123986-86c99dc5d5-7n5p8"] Nov 02 01:25:16.623729 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:16.623705 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-123986-86c99dc5d5-7n5p8" Nov 02 01:25:16.634624 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:16.634591 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-123986-serving-cert\"" Nov 02 01:25:16.634822 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:16.634699 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 01:25:16.634991 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:16.634763 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8qwrd\"" Nov 02 01:25:16.643593 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:16.643569 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-123986-86c99dc5d5-7n5p8"] Nov 02 01:25:16.721492 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:16.721459 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f12da917-8a8f-445b-b939-0bab19110258-metrics-certs\") pod \"maas-api-e2e-worker-w5-123986-86c99dc5d5-7n5p8\" (UID: \"f12da917-8a8f-445b-b939-0bab19110258\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-123986-86c99dc5d5-7n5p8" Nov 02 01:25:16.721708 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:16.721509 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f12da917-8a8f-445b-b939-0bab19110258-maas-api-tls\") pod \"maas-api-e2e-worker-w5-123986-86c99dc5d5-7n5p8\" (UID: \"f12da917-8a8f-445b-b939-0bab19110258\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-123986-86c99dc5d5-7n5p8" Nov 02 01:25:16.721708 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:16.721653 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f12da917-8a8f-445b-b939-0bab19110258-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-123986-86c99dc5d5-7n5p8\" (UID: \"f12da917-8a8f-445b-b939-0bab19110258\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-123986-86c99dc5d5-7n5p8" Nov 02 01:25:16.721807 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:16.721761 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg9p6\" (UniqueName: \"kubernetes.io/projected/f12da917-8a8f-445b-b939-0bab19110258-kube-api-access-kg9p6\") pod \"maas-api-e2e-worker-w5-123986-86c99dc5d5-7n5p8\" (UID: \"f12da917-8a8f-445b-b939-0bab19110258\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-123986-86c99dc5d5-7n5p8" Nov 02 01:25:16.823182 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:16.823127 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f12da917-8a8f-445b-b939-0bab19110258-metrics-certs\") pod \"maas-api-e2e-worker-w5-123986-86c99dc5d5-7n5p8\" (UID: \"f12da917-8a8f-445b-b939-0bab19110258\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-123986-86c99dc5d5-7n5p8" Nov 02 01:25:16.823349 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:16.823201 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f12da917-8a8f-445b-b939-0bab19110258-maas-api-tls\") pod \"maas-api-e2e-worker-w5-123986-86c99dc5d5-7n5p8\" (UID: \"f12da917-8a8f-445b-b939-0bab19110258\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-123986-86c99dc5d5-7n5p8" Nov 02 01:25:16.823349 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:16.823267 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f12da917-8a8f-445b-b939-0bab19110258-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-123986-86c99dc5d5-7n5p8\" (UID: \"f12da917-8a8f-445b-b939-0bab19110258\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-123986-86c99dc5d5-7n5p8" Nov 02 01:25:16.824774 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:16.823349 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg9p6\" (UniqueName: \"kubernetes.io/projected/f12da917-8a8f-445b-b939-0bab19110258-kube-api-access-kg9p6\") pod \"maas-api-e2e-worker-w5-123986-86c99dc5d5-7n5p8\" (UID: \"f12da917-8a8f-445b-b939-0bab19110258\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-123986-86c99dc5d5-7n5p8" Nov 02 01:25:16.824774 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:16.824525 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f12da917-8a8f-445b-b939-0bab19110258-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-123986-86c99dc5d5-7n5p8\" (UID: \"f12da917-8a8f-445b-b939-0bab19110258\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-123986-86c99dc5d5-7n5p8" Nov 02 01:25:16.826925 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:16.826873 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f12da917-8a8f-445b-b939-0bab19110258-maas-api-tls\") pod \"maas-api-e2e-worker-w5-123986-86c99dc5d5-7n5p8\" (UID: \"f12da917-8a8f-445b-b939-0bab19110258\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-123986-86c99dc5d5-7n5p8" Nov 02 01:25:16.827769 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:16.827736 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f12da917-8a8f-445b-b939-0bab19110258-metrics-certs\") pod \"maas-api-e2e-worker-w5-123986-86c99dc5d5-7n5p8\" (UID: \"f12da917-8a8f-445b-b939-0bab19110258\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-123986-86c99dc5d5-7n5p8" Nov 02 01:25:16.836977 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:16.836921 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg9p6\" (UniqueName: \"kubernetes.io/projected/f12da917-8a8f-445b-b939-0bab19110258-kube-api-access-kg9p6\") pod \"maas-api-e2e-worker-w5-123986-86c99dc5d5-7n5p8\" (UID: \"f12da917-8a8f-445b-b939-0bab19110258\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-123986-86c99dc5d5-7n5p8" Nov 02 01:25:16.940359 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:16.940269 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-123986-86c99dc5d5-7n5p8" Nov 02 01:25:17.231487 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:17.231223 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-123986-86c99dc5d5-7n5p8"] Nov 02 01:25:17.234367 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:25:17.234327 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf12da917_8a8f_445b_b939_0bab19110258.slice/crio-0404cbba2f9f05e1288a824169215049d1407db1cb46120797611e797ce28498 WatchSource:0}: Error finding container 0404cbba2f9f05e1288a824169215049d1407db1cb46120797611e797ce28498: Status 404 returned error can't find the container with id 0404cbba2f9f05e1288a824169215049d1407db1cb46120797611e797ce28498 Nov 02 01:25:17.236316 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:17.236293 4427 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 01:25:18.137181 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:18.137147 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b6ec07-d4f775449-gjvzd"] Nov 02 01:25:18.141098 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:18.141063 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-b6ec07-d4f775449-gjvzd" Nov 02 01:25:18.151835 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:18.151802 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b6ec07-d4f775449-gjvzd"] Nov 02 01:25:18.195625 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:18.195598 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-b6ec07\"" Nov 02 01:25:18.195625 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:18.195639 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-b6ec07-dockercfg-j5v8z\"" Nov 02 01:25:18.204447 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:18.204407 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-123986-86c99dc5d5-7n5p8" event={"ID":"f12da917-8a8f-445b-b939-0bab19110258","Type":"ContainerStarted","Data":"e7b24774ee2a5b38c99290040b1d604c87842dd18b5dc58f99e340169859ce05"} Nov 02 01:25:18.204625 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:18.204473 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-123986-86c99dc5d5-7n5p8" event={"ID":"f12da917-8a8f-445b-b939-0bab19110258","Type":"ContainerStarted","Data":"0404cbba2f9f05e1288a824169215049d1407db1cb46120797611e797ce28498"} Nov 02 01:25:18.204625 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:18.204609 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-123986-86c99dc5d5-7n5p8" Nov 02 01:25:18.226191 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:18.226134 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-123986-86c99dc5d5-7n5p8" podStartSLOduration=1.7556066860000001 podStartE2EDuration="2.22611546s" totalImagesPullingTime="470.508774ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:25:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:25:18.225226297 +0000 UTC m=+622.846231509" watchObservedRunningTime="2026-11-02 01:25:18.22611546 +0000 UTC m=+622.847120672" Nov 02 01:25:18.238014 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:18.237983 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ad049fe-74e6-4083-92ed-09db1fb92ada-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-b6ec07-d4f775449-gjvzd\" (UID: \"7ad049fe-74e6-4083-92ed-09db1fb92ada\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b6ec07-d4f775449-gjvzd" Nov 02 01:25:18.238167 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:18.238046 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56m9d\" (UniqueName: \"kubernetes.io/projected/7ad049fe-74e6-4083-92ed-09db1fb92ada-kube-api-access-56m9d\") pod \"payload-processing-e2e-worker-w4-b6ec07-d4f775449-gjvzd\" (UID: \"7ad049fe-74e6-4083-92ed-09db1fb92ada\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b6ec07-d4f775449-gjvzd" Nov 02 01:25:18.338788 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:18.338750 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56m9d\" (UniqueName: \"kubernetes.io/projected/7ad049fe-74e6-4083-92ed-09db1fb92ada-kube-api-access-56m9d\") pod \"payload-processing-e2e-worker-w4-b6ec07-d4f775449-gjvzd\" (UID: \"7ad049fe-74e6-4083-92ed-09db1fb92ada\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b6ec07-d4f775449-gjvzd" Nov 02 01:25:18.338963 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:18.338873 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ad049fe-74e6-4083-92ed-09db1fb92ada-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-b6ec07-d4f775449-gjvzd\" (UID: \"7ad049fe-74e6-4083-92ed-09db1fb92ada\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b6ec07-d4f775449-gjvzd" Nov 02 01:25:18.339611 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:18.339589 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ad049fe-74e6-4083-92ed-09db1fb92ada-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-b6ec07-d4f775449-gjvzd\" (UID: \"7ad049fe-74e6-4083-92ed-09db1fb92ada\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b6ec07-d4f775449-gjvzd" Nov 02 01:25:18.351387 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:18.351339 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56m9d\" (UniqueName: \"kubernetes.io/projected/7ad049fe-74e6-4083-92ed-09db1fb92ada-kube-api-access-56m9d\") pod \"payload-processing-e2e-worker-w4-b6ec07-d4f775449-gjvzd\" (UID: \"7ad049fe-74e6-4083-92ed-09db1fb92ada\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b6ec07-d4f775449-gjvzd" Nov 02 01:25:18.459673 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:18.459580 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-b6ec07-d4f775449-gjvzd" Nov 02 01:25:18.799366 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:18.799328 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b6ec07-d4f775449-gjvzd"] Nov 02 01:25:18.801790 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:25:18.801764 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ad049fe_74e6_4083_92ed_09db1fb92ada.slice/crio-b0c613d56bd797022ab4b7a8cb20dc381dcddf26fcb6e90a351c1d9a0ce39f2e WatchSource:0}: Error finding container b0c613d56bd797022ab4b7a8cb20dc381dcddf26fcb6e90a351c1d9a0ce39f2e: Status 404 returned error can't find the container with id b0c613d56bd797022ab4b7a8cb20dc381dcddf26fcb6e90a351c1d9a0ce39f2e Nov 02 01:25:19.210204 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:19.210168 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-b6ec07-d4f775449-gjvzd" event={"ID":"7ad049fe-74e6-4083-92ed-09db1fb92ada","Type":"ContainerStarted","Data":"b0c613d56bd797022ab4b7a8cb20dc381dcddf26fcb6e90a351c1d9a0ce39f2e"} Nov 02 01:25:19.633571 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:19.633121 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ca1c53-cff6c4869-cmlfp"] Nov 02 01:25:19.640052 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:19.640021 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ca1c53-cff6c4869-cmlfp" Nov 02 01:25:19.646921 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:19.646412 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-ca1c53-serving-cert\"" Nov 02 01:25:19.649801 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:19.649747 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ca1c53-cff6c4869-cmlfp"] Nov 02 01:25:19.752200 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:19.752152 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe3cde15-cd40-4b6a-9144-f96a05179a27-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-ca1c53-cff6c4869-cmlfp\" (UID: \"fe3cde15-cd40-4b6a-9144-f96a05179a27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ca1c53-cff6c4869-cmlfp" Nov 02 01:25:19.752361 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:19.752317 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe3cde15-cd40-4b6a-9144-f96a05179a27-metrics-certs\") pod \"maas-api-e2e-worker-w0-ca1c53-cff6c4869-cmlfp\" (UID: \"fe3cde15-cd40-4b6a-9144-f96a05179a27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ca1c53-cff6c4869-cmlfp" Nov 02 01:25:19.752439 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:19.752360 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe3cde15-cd40-4b6a-9144-f96a05179a27-maas-api-tls\") pod \"maas-api-e2e-worker-w0-ca1c53-cff6c4869-cmlfp\" (UID: \"fe3cde15-cd40-4b6a-9144-f96a05179a27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ca1c53-cff6c4869-cmlfp" Nov 02 01:25:19.752481 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:19.752454 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv58v\" (UniqueName: \"kubernetes.io/projected/fe3cde15-cd40-4b6a-9144-f96a05179a27-kube-api-access-fv58v\") pod \"maas-api-e2e-worker-w0-ca1c53-cff6c4869-cmlfp\" (UID: \"fe3cde15-cd40-4b6a-9144-f96a05179a27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ca1c53-cff6c4869-cmlfp" Nov 02 01:25:19.853926 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:19.853886 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe3cde15-cd40-4b6a-9144-f96a05179a27-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-ca1c53-cff6c4869-cmlfp\" (UID: \"fe3cde15-cd40-4b6a-9144-f96a05179a27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ca1c53-cff6c4869-cmlfp" Nov 02 01:25:19.854097 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:19.854001 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe3cde15-cd40-4b6a-9144-f96a05179a27-metrics-certs\") pod \"maas-api-e2e-worker-w0-ca1c53-cff6c4869-cmlfp\" (UID: \"fe3cde15-cd40-4b6a-9144-f96a05179a27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ca1c53-cff6c4869-cmlfp" Nov 02 01:25:19.854097 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:19.854036 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe3cde15-cd40-4b6a-9144-f96a05179a27-maas-api-tls\") pod \"maas-api-e2e-worker-w0-ca1c53-cff6c4869-cmlfp\" (UID: \"fe3cde15-cd40-4b6a-9144-f96a05179a27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ca1c53-cff6c4869-cmlfp" Nov 02 01:25:19.854097 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:19.854081 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv58v\" (UniqueName: \"kubernetes.io/projected/fe3cde15-cd40-4b6a-9144-f96a05179a27-kube-api-access-fv58v\") pod \"maas-api-e2e-worker-w0-ca1c53-cff6c4869-cmlfp\" (UID: \"fe3cde15-cd40-4b6a-9144-f96a05179a27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ca1c53-cff6c4869-cmlfp" Nov 02 01:25:19.855133 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:19.855103 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe3cde15-cd40-4b6a-9144-f96a05179a27-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-ca1c53-cff6c4869-cmlfp\" (UID: \"fe3cde15-cd40-4b6a-9144-f96a05179a27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ca1c53-cff6c4869-cmlfp" Nov 02 01:25:19.857918 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:19.857892 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe3cde15-cd40-4b6a-9144-f96a05179a27-metrics-certs\") pod \"maas-api-e2e-worker-w0-ca1c53-cff6c4869-cmlfp\" (UID: \"fe3cde15-cd40-4b6a-9144-f96a05179a27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ca1c53-cff6c4869-cmlfp" Nov 02 01:25:19.858043 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:19.858021 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe3cde15-cd40-4b6a-9144-f96a05179a27-maas-api-tls\") pod \"maas-api-e2e-worker-w0-ca1c53-cff6c4869-cmlfp\" (UID: \"fe3cde15-cd40-4b6a-9144-f96a05179a27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ca1c53-cff6c4869-cmlfp" Nov 02 01:25:19.889456 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:19.889145 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv58v\" (UniqueName: \"kubernetes.io/projected/fe3cde15-cd40-4b6a-9144-f96a05179a27-kube-api-access-fv58v\") pod \"maas-api-e2e-worker-w0-ca1c53-cff6c4869-cmlfp\" (UID: \"fe3cde15-cd40-4b6a-9144-f96a05179a27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ca1c53-cff6c4869-cmlfp" Nov 02 01:25:19.986398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:19.985674 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ca1c53-cff6c4869-cmlfp" Nov 02 01:25:20.309680 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:25:20.309645 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe3cde15_cd40_4b6a_9144_f96a05179a27.slice/crio-a2cda749dded8af96f29c25917936829587cd7c4dd2771479b657c5cc6a9a309 WatchSource:0}: Error finding container a2cda749dded8af96f29c25917936829587cd7c4dd2771479b657c5cc6a9a309: Status 404 returned error can't find the container with id a2cda749dded8af96f29c25917936829587cd7c4dd2771479b657c5cc6a9a309 Nov 02 01:25:20.310101 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:20.309687 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ca1c53-cff6c4869-cmlfp"] Nov 02 01:25:21.223702 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:21.223667 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ca1c53-cff6c4869-cmlfp" event={"ID":"fe3cde15-cd40-4b6a-9144-f96a05179a27","Type":"ContainerStarted","Data":"779cc2d78aedeb795aa0d2b16ed8694560b30d293757242756ab50e24d250587"} Nov 02 01:25:21.223846 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:21.223723 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ca1c53-cff6c4869-cmlfp" event={"ID":"fe3cde15-cd40-4b6a-9144-f96a05179a27","Type":"ContainerStarted","Data":"a2cda749dded8af96f29c25917936829587cd7c4dd2771479b657c5cc6a9a309"} Nov 02 01:25:21.224396 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:21.224341 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ca1c53-cff6c4869-cmlfp" Nov 02 01:25:22.237346 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:22.237297 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-b6ec07-d4f775449-gjvzd" event={"ID":"7ad049fe-74e6-4083-92ed-09db1fb92ada","Type":"ContainerStarted","Data":"d3363e7b99cf7a6a00ff13e395b4cf3a8c0c978d0dd1e825bf57d06593382b88"} Nov 02 01:25:22.237830 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:22.237572 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-b6ec07-d4f775449-gjvzd" Nov 02 01:25:22.288818 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:22.288690 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ca1c53-cff6c4869-cmlfp" podStartSLOduration=2.786555935 podStartE2EDuration="3.288672842s" totalImagesPullingTime="502.116907ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:25:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:25:21.244566862 +0000 UTC m=+625.865572072" watchObservedRunningTime="2026-11-02 01:25:22.288672842 +0000 UTC m=+626.909678054" Nov 02 01:25:22.289153 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:22.289117 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-b6ec07-d4f775449-gjvzd" podStartSLOduration=1.069685984 podStartE2EDuration="4.289106142s" totalImagesPullingTime="3.219420158s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:25:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:25:22.288611287 +0000 UTC m=+626.909616499" watchObservedRunningTime="2026-11-02 01:25:22.289106142 +0000 UTC m=+626.910111354" Nov 02 01:25:24.245971 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:24.245601 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-123986-86c99dc5d5-7n5p8" Nov 02 01:25:27.214402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:27.212545 4427 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 01:25:27.214402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:27.212612 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf" podUID="b880c081-4957-4151-9e1f-fb8f397dedb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:25:28.249996 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:28.249971 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ca1c53-cff6c4869-cmlfp" Nov 02 01:25:30.991485 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:30.989347 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk"] Nov 02 01:25:30.993731 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:30.993696 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" Nov 02 01:25:31.001035 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:31.001005 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-123986\"/\"openshift-service-ca.crt\"" Nov 02 01:25:31.002572 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:31.002460 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-123986\"/\"kube-root-ca.crt\"" Nov 02 01:25:31.002572 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:31.002525 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-123986\"/\"default-dockercfg-fhs95\"" Nov 02 01:25:31.002777 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:31.002601 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-123986\"/\"simulator-w5-123986-kserve-self-signed-certs\"" Nov 02 01:25:31.007738 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:31.007225 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk"] Nov 02 01:25:31.075928 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:31.075838 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsbd7\" (UniqueName: \"kubernetes.io/projected/d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb-kube-api-access-lsbd7\") pod \"simulator-w5-123986-kserve-65dfb74c9-k5tzk\" (UID: \"d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb\") " pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" Nov 02 01:25:31.076161 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:31.075931 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb-home\") pod \"simulator-w5-123986-kserve-65dfb74c9-k5tzk\" (UID: \"d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb\") " pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" Nov 02 01:25:31.076161 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:31.075975 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb-tls-certs\") pod \"simulator-w5-123986-kserve-65dfb74c9-k5tzk\" (UID: \"d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb\") " pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" Nov 02 01:25:31.076161 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:31.076031 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb-dshm\") pod \"simulator-w5-123986-kserve-65dfb74c9-k5tzk\" (UID: \"d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb\") " pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" Nov 02 01:25:31.076161 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:31.076054 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb-model-cache\") pod \"simulator-w5-123986-kserve-65dfb74c9-k5tzk\" (UID: \"d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb\") " pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" Nov 02 01:25:31.076161 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:31.076089 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb-tmp-dir\") pod \"simulator-w5-123986-kserve-65dfb74c9-k5tzk\" (UID: \"d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb\") " pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" Nov 02 01:25:31.179642 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:31.177348 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb-tmp-dir\") pod \"simulator-w5-123986-kserve-65dfb74c9-k5tzk\" (UID: \"d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb\") " pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" Nov 02 01:25:31.179642 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:31.177423 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsbd7\" (UniqueName: \"kubernetes.io/projected/d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb-kube-api-access-lsbd7\") pod \"simulator-w5-123986-kserve-65dfb74c9-k5tzk\" (UID: \"d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb\") " pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" Nov 02 01:25:31.179642 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:31.177487 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb-home\") pod \"simulator-w5-123986-kserve-65dfb74c9-k5tzk\" (UID: \"d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb\") " pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" Nov 02 01:25:31.179642 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:31.177546 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb-tls-certs\") pod \"simulator-w5-123986-kserve-65dfb74c9-k5tzk\" (UID: \"d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb\") " pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" Nov 02 01:25:31.179642 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:31.177634 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb-dshm\") pod \"simulator-w5-123986-kserve-65dfb74c9-k5tzk\" (UID: \"d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb\") " pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" Nov 02 01:25:31.179642 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:31.177668 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb-model-cache\") pod \"simulator-w5-123986-kserve-65dfb74c9-k5tzk\" (UID: \"d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb\") " pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" Nov 02 01:25:31.179642 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:31.178140 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb-model-cache\") pod \"simulator-w5-123986-kserve-65dfb74c9-k5tzk\" (UID: \"d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb\") " pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" Nov 02 01:25:31.179642 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:31.178412 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb-tmp-dir\") pod \"simulator-w5-123986-kserve-65dfb74c9-k5tzk\" (UID: \"d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb\") " pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" Nov 02 01:25:31.179642 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:31.178977 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb-home\") pod \"simulator-w5-123986-kserve-65dfb74c9-k5tzk\" (UID: \"d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb\") " pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" Nov 02 01:25:31.182296 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:31.182243 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb-tls-certs\") pod \"simulator-w5-123986-kserve-65dfb74c9-k5tzk\" (UID: \"d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb\") " pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" Nov 02 01:25:31.183155 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:31.183089 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb-dshm\") pod \"simulator-w5-123986-kserve-65dfb74c9-k5tzk\" (UID: \"d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb\") " pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" Nov 02 01:25:31.193612 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:31.193548 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsbd7\" (UniqueName: \"kubernetes.io/projected/d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb-kube-api-access-lsbd7\") pod \"simulator-w5-123986-kserve-65dfb74c9-k5tzk\" (UID: \"d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb\") " pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" Nov 02 01:25:31.315195 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:31.315155 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" Nov 02 01:25:31.353504 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:31.353468 4427 generic.go:356] "Generic (PLEG): container finished" podID="b880c081-4957-4151-9e1f-fb8f397dedb3" containerID="b5107dd9a7488440da3a86139d85e7887cda8bd9a4f18ed1e23170d1d29f81d0" exitCode=0 Nov 02 01:25:31.353664 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:31.353539 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf" event={"ID":"b880c081-4957-4151-9e1f-fb8f397dedb3","Type":"ContainerDied","Data":"b5107dd9a7488440da3a86139d85e7887cda8bd9a4f18ed1e23170d1d29f81d0"} Nov 02 01:25:31.354056 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:31.354041 4427 scope.go:122] "RemoveContainer" containerID="b5107dd9a7488440da3a86139d85e7887cda8bd9a4f18ed1e23170d1d29f81d0" Nov 02 01:25:31.355684 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:31.355645 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:25:31.356178 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:31.355736 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:25:31.356178 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:31.355785 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:25:31.585796 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:25:31.585711 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:25:31.611140 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:31.611099 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk"] Nov 02 01:25:31.621777 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:25:31.621751 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4ae0b1f_50f6_46ff_8a9c_0dc3216265cb.slice/crio-cbe1229b744d9aeac6e0cdef8ab2da1bc540bf75d7c37bcbd5008fb295a3a66d WatchSource:0}: Error finding container cbe1229b744d9aeac6e0cdef8ab2da1bc540bf75d7c37bcbd5008fb295a3a66d: Status 404 returned error can't find the container with id cbe1229b744d9aeac6e0cdef8ab2da1bc540bf75d7c37bcbd5008fb295a3a66d Nov 02 01:25:32.391289 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:32.391005 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" event={"ID":"d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb","Type":"ContainerStarted","Data":"625b727e7e64bfbe62ecb5c63f32f277e94d73e9a697a5f7dfae9d0227ffd763"} Nov 02 01:25:32.391289 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:32.391063 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" event={"ID":"d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb","Type":"ContainerStarted","Data":"cbe1229b744d9aeac6e0cdef8ab2da1bc540bf75d7c37bcbd5008fb295a3a66d"} Nov 02 01:25:32.394911 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:32.393807 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf" event={"ID":"b880c081-4957-4151-9e1f-fb8f397dedb3","Type":"ContainerStarted","Data":"16e18f9cf7042ff833baa19cc64cabba064002c9fb876efdd1927e2c6e07a2a8"} Nov 02 01:25:32.413955 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:32.413884 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" podStartSLOduration=2.413866979 podStartE2EDuration="2.413866979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:25:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:25:32.411979665 +0000 UTC m=+637.032984889" watchObservedRunningTime="2026-11-02 01:25:32.413866979 +0000 UTC m=+637.034872184" Nov 02 01:25:33.227612 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:33.227568 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf" Nov 02 01:25:33.227859 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:33.227837 4427 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 02 01:25:33.227946 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:33.227890 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf" podUID="b880c081-4957-4151-9e1f-fb8f397dedb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 02 01:25:33.253233 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:33.252425 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-b6ec07-d4f775449-gjvzd" Nov 02 01:25:33.320238 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:33.320179 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" Nov 02 01:25:34.227641 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:34.227602 4427 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 02 01:25:34.228153 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:34.227684 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf" podUID="b880c081-4957-4151-9e1f-fb8f397dedb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 02 01:25:34.427397 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:34.427341 4427 generic.go:356] "Generic (PLEG): container finished" podID="282522e6-7d16-4316-bf07-5e2706155435" containerID="3e4e3e4518a25da89666056534b7618c728efc423e571f4d498aa3205516d6ef" exitCode=0 Nov 02 01:25:34.427568 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:34.427462 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl" event={"ID":"282522e6-7d16-4316-bf07-5e2706155435","Type":"ContainerDied","Data":"3e4e3e4518a25da89666056534b7618c728efc423e571f4d498aa3205516d6ef"} Nov 02 01:25:34.427830 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:34.427812 4427 scope.go:122] "RemoveContainer" containerID="3e4e3e4518a25da89666056534b7618c728efc423e571f4d498aa3205516d6ef" Nov 02 01:25:34.429762 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:34.429736 4427 generic.go:356] "Generic (PLEG): container finished" podID="19218597-2356-4f9d-bf2b-ac4597da575a" containerID="6d73244c1f5558e36ec4b736e792c4ed38240123737cf0dd0ccd5bf8a0089be7" exitCode=0 Nov 02 01:25:34.429877 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:34.429826 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9" event={"ID":"19218597-2356-4f9d-bf2b-ac4597da575a","Type":"ContainerDied","Data":"6d73244c1f5558e36ec4b736e792c4ed38240123737cf0dd0ccd5bf8a0089be7"} Nov 02 01:25:34.430191 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:34.430171 4427 scope.go:122] "RemoveContainer" containerID="6d73244c1f5558e36ec4b736e792c4ed38240123737cf0dd0ccd5bf8a0089be7" Nov 02 01:25:34.430275 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:34.430223 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:25:34.430362 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:34.430343 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:25:34.430466 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:34.430445 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:25:34.431529 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:34.431504 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:25:34.431639 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:34.431574 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:25:34.431703 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:34.431686 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:25:35.231779 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:35.231735 4427 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 02 01:25:35.232263 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:35.231790 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf" podUID="b880c081-4957-4151-9e1f-fb8f397dedb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 02 01:25:35.442319 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:35.442283 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl" event={"ID":"282522e6-7d16-4316-bf07-5e2706155435","Type":"ContainerStarted","Data":"3fc39fb8b747ca34e53fcc1003d509ec7f0fcdaa6cbf66eeccf80d36e47caaa5"} Nov 02 01:25:35.447616 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:35.447547 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9" event={"ID":"19218597-2356-4f9d-bf2b-ac4597da575a","Type":"ContainerStarted","Data":"93aae06e287d80ec36542db437424297b58b59e34de95c9a40a02a948dfc96d7"} Nov 02 01:25:36.049708 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:36.049669 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl" Nov 02 01:25:36.050008 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:36.049987 4427 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 02 01:25:36.050054 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:36.050036 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl" podUID="282522e6-7d16-4316-bf07-5e2706155435" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 02 01:25:36.228401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:36.227823 4427 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 02 01:25:36.228401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:36.227887 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf" podUID="b880c081-4957-4151-9e1f-fb8f397dedb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 02 01:25:36.421400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:36.420750 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9" Nov 02 01:25:36.421400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:36.420869 4427 patch_prober.go:29] interesting pod/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 02 01:25:36.421400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:36.420911 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9" podUID="19218597-2356-4f9d-bf2b-ac4597da575a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 02 01:25:37.050239 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:37.049761 4427 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 02 01:25:37.050239 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:37.049834 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl" podUID="282522e6-7d16-4316-bf07-5e2706155435" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 02 01:25:37.240399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:37.239863 4427 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 02 01:25:37.240399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:37.239947 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf" podUID="b880c081-4957-4151-9e1f-fb8f397dedb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 02 01:25:37.423399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:37.422816 4427 patch_prober.go:29] interesting pod/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 02 01:25:37.423399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:37.422890 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9" podUID="19218597-2356-4f9d-bf2b-ac4597da575a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 02 01:25:38.049948 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:38.049912 4427 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 02 01:25:38.050125 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:38.049970 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl" podUID="282522e6-7d16-4316-bf07-5e2706155435" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 02 01:25:38.423052 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:38.422988 4427 patch_prober.go:29] interesting pod/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 02 01:25:38.423335 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:38.423057 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9" podUID="19218597-2356-4f9d-bf2b-ac4597da575a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 02 01:25:38.652122 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:38.652091 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf" Nov 02 01:25:38.652999 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:38.652981 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf" Nov 02 01:25:38.658759 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:38.658738 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf" Nov 02 01:25:38.850542 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:38.850452 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t"] Nov 02 01:25:38.860591 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:38.860561 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t"] Nov 02 01:25:38.868625 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:38.860907 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" Nov 02 01:25:38.936547 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:38.936514 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-b9d816\"/\"openshift-service-ca.crt\"" Nov 02 01:25:38.937290 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:38.937267 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-b9d816\"/\"kube-root-ca.crt\"" Nov 02 01:25:38.937826 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:38.937811 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-b9d816\"/\"simulator-w6-b9d816-kserve-self-signed-certs\"" Nov 02 01:25:38.937992 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:38.937977 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-b9d816\"/\"default-dockercfg-7z7ft\"" Nov 02 01:25:38.980603 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:38.980235 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189-dshm\") pod \"simulator-w6-b9d816-kserve-57db454ff-sd28t\" (UID: \"eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189\") " pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" Nov 02 01:25:38.980603 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:38.980354 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189-tmp-dir\") pod \"simulator-w6-b9d816-kserve-57db454ff-sd28t\" (UID: \"eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189\") " pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" Nov 02 01:25:38.980603 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:38.980434 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189-tls-certs\") pod \"simulator-w6-b9d816-kserve-57db454ff-sd28t\" (UID: \"eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189\") " pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" Nov 02 01:25:38.980603 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:38.980462 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnfmd\" (UniqueName: \"kubernetes.io/projected/eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189-kube-api-access-vnfmd\") pod \"simulator-w6-b9d816-kserve-57db454ff-sd28t\" (UID: \"eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189\") " pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" Nov 02 01:25:38.980603 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:38.980514 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189-home\") pod \"simulator-w6-b9d816-kserve-57db454ff-sd28t\" (UID: \"eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189\") " pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" Nov 02 01:25:38.980603 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:38.980566 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189-model-cache\") pod \"simulator-w6-b9d816-kserve-57db454ff-sd28t\" (UID: \"eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189\") " pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" Nov 02 01:25:39.050108 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:39.050068 4427 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 02 01:25:39.050308 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:39.050136 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl" podUID="282522e6-7d16-4316-bf07-5e2706155435" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 02 01:25:39.087351 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:39.087302 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189-tmp-dir\") pod \"simulator-w6-b9d816-kserve-57db454ff-sd28t\" (UID: \"eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189\") " pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" Nov 02 01:25:39.088285 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:39.088262 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189-tls-certs\") pod \"simulator-w6-b9d816-kserve-57db454ff-sd28t\" (UID: \"eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189\") " pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" Nov 02 01:25:39.088513 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:39.088494 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnfmd\" (UniqueName: \"kubernetes.io/projected/eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189-kube-api-access-vnfmd\") pod \"simulator-w6-b9d816-kserve-57db454ff-sd28t\" (UID: \"eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189\") " pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" Nov 02 01:25:39.089669 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:39.089648 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189-home\") pod \"simulator-w6-b9d816-kserve-57db454ff-sd28t\" (UID: \"eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189\") " pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" Nov 02 01:25:39.090217 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:39.090196 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189-model-cache\") pod \"simulator-w6-b9d816-kserve-57db454ff-sd28t\" (UID: \"eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189\") " pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" Nov 02 01:25:39.090768 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:39.088156 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189-tmp-dir\") pod \"simulator-w6-b9d816-kserve-57db454ff-sd28t\" (UID: \"eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189\") " pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" Nov 02 01:25:39.091246 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:39.090648 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189-model-cache\") pod \"simulator-w6-b9d816-kserve-57db454ff-sd28t\" (UID: \"eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189\") " pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" Nov 02 01:25:39.091430 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:39.090076 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189-home\") pod \"simulator-w6-b9d816-kserve-57db454ff-sd28t\" (UID: \"eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189\") " pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" Nov 02 01:25:39.091568 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:39.090732 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189-dshm\") pod \"simulator-w6-b9d816-kserve-57db454ff-sd28t\" (UID: \"eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189\") " pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" Nov 02 01:25:39.111218 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:39.111184 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnfmd\" (UniqueName: \"kubernetes.io/projected/eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189-kube-api-access-vnfmd\") pod \"simulator-w6-b9d816-kserve-57db454ff-sd28t\" (UID: \"eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189\") " pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" Nov 02 01:25:39.132364 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:39.126077 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189-tls-certs\") pod \"simulator-w6-b9d816-kserve-57db454ff-sd28t\" (UID: \"eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189\") " pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" Nov 02 01:25:39.132973 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:39.132894 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189-dshm\") pod \"simulator-w6-b9d816-kserve-57db454ff-sd28t\" (UID: \"eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189\") " pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" Nov 02 01:25:39.232414 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:39.232094 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" Nov 02 01:25:39.434150 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:39.433754 4427 patch_prober.go:29] interesting pod/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 02 01:25:39.434150 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:39.433813 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9" podUID="19218597-2356-4f9d-bf2b-ac4597da575a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 02 01:25:39.625780 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:39.616950 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t"] Nov 02 01:25:39.625780 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:25:39.624962 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeea5c69d_8cda_4fa3_a0d1_dbc2ab6cf189.slice/crio-a2283896984978451d1ad9039bb8831847adeda065e7487a32f147af624ff6fd WatchSource:0}: Error finding container a2283896984978451d1ad9039bb8831847adeda065e7487a32f147af624ff6fd: Status 404 returned error can't find the container with id a2283896984978451d1ad9039bb8831847adeda065e7487a32f147af624ff6fd Nov 02 01:25:40.050019 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:40.049713 4427 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 02 01:25:40.050019 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:40.049775 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl" podUID="282522e6-7d16-4316-bf07-5e2706155435" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 02 01:25:40.420977 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:40.420946 4427 patch_prober.go:29] interesting pod/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 02 01:25:40.421143 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:40.420998 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9" podUID="19218597-2356-4f9d-bf2b-ac4597da575a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 02 01:25:40.538011 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:40.537972 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" event={"ID":"eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189","Type":"ContainerStarted","Data":"027a348e1131304bdf6ccde156cb01bda7fa1239b9c9fad3a6022508c50d43e4"} Nov 02 01:25:40.538175 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:40.538028 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" event={"ID":"eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189","Type":"ContainerStarted","Data":"a2283896984978451d1ad9039bb8831847adeda065e7487a32f147af624ff6fd"} Nov 02 01:25:41.050299 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:41.050259 4427 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 02 01:25:41.050847 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:41.050326 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl" podUID="282522e6-7d16-4316-bf07-5e2706155435" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 02 01:25:41.167059 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:41.167018 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9" Nov 02 01:25:41.167242 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:41.167134 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl" Nov 02 01:25:41.233490 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:41.233462 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" Nov 02 01:25:41.316155 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:41.316059 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" Nov 02 01:25:41.329686 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:41.329651 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" Nov 02 01:25:41.353527 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:41.353467 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" podStartSLOduration=3.353447331 podStartE2EDuration="3.353447331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:25:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:25:40.559366059 +0000 UTC m=+645.180371267" watchObservedRunningTime="2026-11-02 01:25:41.353447331 +0000 UTC m=+645.974452547" Nov 02 01:25:41.425881 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:41.425850 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9" Nov 02 01:25:41.550560 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:41.550527 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9" Nov 02 01:25:41.565454 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:41.565427 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" Nov 02 01:25:42.053900 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:42.053870 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl" Nov 02 01:25:42.055013 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:42.054989 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl" Nov 02 01:25:46.749828 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:46.749795 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm"] Nov 02 01:25:46.756404 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:46.756359 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" Nov 02 01:25:46.763096 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:46.763048 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5687bc\"/\"openshift-service-ca.crt\"" Nov 02 01:25:46.763416 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:46.763366 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5687bc\"/\"kube-root-ca.crt\"" Nov 02 01:25:46.763620 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:46.763520 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm"] Nov 02 01:25:46.763882 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:46.763861 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5687bc\"/\"simulator-w1-5687bc-kserve-self-signed-certs\"" Nov 02 01:25:46.763882 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:46.763874 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5687bc\"/\"default-dockercfg-qz5qt\"" Nov 02 01:25:46.812692 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:46.812654 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb533b5f-3525-454a-8ce2-ea9246194f28-tls-certs\") pod \"simulator-w1-5687bc-kserve-75d664fc67-gclhm\" (UID: \"fb533b5f-3525-454a-8ce2-ea9246194f28\") " pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" Nov 02 01:25:46.812936 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:46.812719 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb533b5f-3525-454a-8ce2-ea9246194f28-model-cache\") pod \"simulator-w1-5687bc-kserve-75d664fc67-gclhm\" (UID: \"fb533b5f-3525-454a-8ce2-ea9246194f28\") " pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" Nov 02 01:25:46.812936 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:46.812840 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb533b5f-3525-454a-8ce2-ea9246194f28-dshm\") pod \"simulator-w1-5687bc-kserve-75d664fc67-gclhm\" (UID: \"fb533b5f-3525-454a-8ce2-ea9246194f28\") " pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" Nov 02 01:25:46.812936 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:46.812884 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb533b5f-3525-454a-8ce2-ea9246194f28-tmp-dir\") pod \"simulator-w1-5687bc-kserve-75d664fc67-gclhm\" (UID: \"fb533b5f-3525-454a-8ce2-ea9246194f28\") " pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" Nov 02 01:25:46.813084 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:46.812967 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb533b5f-3525-454a-8ce2-ea9246194f28-home\") pod \"simulator-w1-5687bc-kserve-75d664fc67-gclhm\" (UID: \"fb533b5f-3525-454a-8ce2-ea9246194f28\") " pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" Nov 02 01:25:46.813084 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:46.812994 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74g7g\" (UniqueName: \"kubernetes.io/projected/fb533b5f-3525-454a-8ce2-ea9246194f28-kube-api-access-74g7g\") pod \"simulator-w1-5687bc-kserve-75d664fc67-gclhm\" (UID: \"fb533b5f-3525-454a-8ce2-ea9246194f28\") " pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" Nov 02 01:25:46.914399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:46.914337 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb533b5f-3525-454a-8ce2-ea9246194f28-model-cache\") pod \"simulator-w1-5687bc-kserve-75d664fc67-gclhm\" (UID: \"fb533b5f-3525-454a-8ce2-ea9246194f28\") " pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" Nov 02 01:25:46.914696 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:46.914675 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb533b5f-3525-454a-8ce2-ea9246194f28-dshm\") pod \"simulator-w1-5687bc-kserve-75d664fc67-gclhm\" (UID: \"fb533b5f-3525-454a-8ce2-ea9246194f28\") " pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" Nov 02 01:25:46.914811 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:46.914798 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb533b5f-3525-454a-8ce2-ea9246194f28-tmp-dir\") pod \"simulator-w1-5687bc-kserve-75d664fc67-gclhm\" (UID: \"fb533b5f-3525-454a-8ce2-ea9246194f28\") " pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" Nov 02 01:25:46.914911 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:46.914898 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb533b5f-3525-454a-8ce2-ea9246194f28-home\") pod \"simulator-w1-5687bc-kserve-75d664fc67-gclhm\" (UID: \"fb533b5f-3525-454a-8ce2-ea9246194f28\") " pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" Nov 02 01:25:46.914999 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:46.914986 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74g7g\" (UniqueName: \"kubernetes.io/projected/fb533b5f-3525-454a-8ce2-ea9246194f28-kube-api-access-74g7g\") pod \"simulator-w1-5687bc-kserve-75d664fc67-gclhm\" (UID: \"fb533b5f-3525-454a-8ce2-ea9246194f28\") " pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" Nov 02 01:25:46.915108 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:46.915062 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb533b5f-3525-454a-8ce2-ea9246194f28-model-cache\") pod \"simulator-w1-5687bc-kserve-75d664fc67-gclhm\" (UID: \"fb533b5f-3525-454a-8ce2-ea9246194f28\") " pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" Nov 02 01:25:46.915264 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:46.915248 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb533b5f-3525-454a-8ce2-ea9246194f28-tls-certs\") pod \"simulator-w1-5687bc-kserve-75d664fc67-gclhm\" (UID: \"fb533b5f-3525-454a-8ce2-ea9246194f28\") " pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" Nov 02 01:25:46.915694 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:46.915668 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb533b5f-3525-454a-8ce2-ea9246194f28-tmp-dir\") pod \"simulator-w1-5687bc-kserve-75d664fc67-gclhm\" (UID: \"fb533b5f-3525-454a-8ce2-ea9246194f28\") " pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" Nov 02 01:25:46.915791 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:46.915737 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb533b5f-3525-454a-8ce2-ea9246194f28-home\") pod \"simulator-w1-5687bc-kserve-75d664fc67-gclhm\" (UID: \"fb533b5f-3525-454a-8ce2-ea9246194f28\") " pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" Nov 02 01:25:46.919281 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:46.919219 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb533b5f-3525-454a-8ce2-ea9246194f28-tls-certs\") pod \"simulator-w1-5687bc-kserve-75d664fc67-gclhm\" (UID: \"fb533b5f-3525-454a-8ce2-ea9246194f28\") " pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" Nov 02 01:25:46.938676 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:46.938637 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb533b5f-3525-454a-8ce2-ea9246194f28-dshm\") pod \"simulator-w1-5687bc-kserve-75d664fc67-gclhm\" (UID: \"fb533b5f-3525-454a-8ce2-ea9246194f28\") " pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" Nov 02 01:25:46.947310 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:46.947280 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74g7g\" (UniqueName: \"kubernetes.io/projected/fb533b5f-3525-454a-8ce2-ea9246194f28-kube-api-access-74g7g\") pod \"simulator-w1-5687bc-kserve-75d664fc67-gclhm\" (UID: \"fb533b5f-3525-454a-8ce2-ea9246194f28\") " pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" Nov 02 01:25:47.086436 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:47.086335 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" Nov 02 01:25:47.365902 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:47.365870 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm"] Nov 02 01:25:47.368384 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:25:47.368342 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb533b5f_3525_454a_8ce2_ea9246194f28.slice/crio-8de130c26b61edd8e7cc11278fa430ed16b0a0b6c68153b2370aafbd28c3e777 WatchSource:0}: Error finding container 8de130c26b61edd8e7cc11278fa430ed16b0a0b6c68153b2370aafbd28c3e777: Status 404 returned error can't find the container with id 8de130c26b61edd8e7cc11278fa430ed16b0a0b6c68153b2370aafbd28c3e777 Nov 02 01:25:47.580416 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:47.580348 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" event={"ID":"fb533b5f-3525-454a-8ce2-ea9246194f28","Type":"ContainerStarted","Data":"17ce9f7ce5468ae59b54a5179dda3e4bc18154a3422a8050dc36150a36580289"} Nov 02 01:25:47.580599 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:47.580419 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" event={"ID":"fb533b5f-3525-454a-8ce2-ea9246194f28","Type":"ContainerStarted","Data":"8de130c26b61edd8e7cc11278fa430ed16b0a0b6c68153b2370aafbd28c3e777"} Nov 02 01:25:47.599625 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:47.599575 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" podStartSLOduration=1.5995617530000001 podStartE2EDuration="1.599561753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:25:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:25:47.598455132 +0000 UTC m=+652.219460343" watchObservedRunningTime="2026-11-02 01:25:47.599561753 +0000 UTC m=+652.220566964" Nov 02 01:25:48.086817 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:48.086779 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" Nov 02 01:25:49.233110 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:49.233077 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" Nov 02 01:25:49.246729 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:49.246579 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" Nov 02 01:25:49.603809 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:49.603722 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" Nov 02 01:25:53.781485 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:53.781452 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn"] Nov 02 01:25:53.787727 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:53.787700 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" Nov 02 01:25:53.794269 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:53.794242 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-123986\"/\"premium-w5-123986-kserve-self-signed-certs\"" Nov 02 01:25:53.798095 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:53.798072 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn"] Nov 02 01:25:53.893606 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:53.893568 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7085c0ab-7ee3-4665-abad-2977c25f68ba-model-cache\") pod \"premium-w5-123986-kserve-6d567889d6-fc7cn\" (UID: \"7085c0ab-7ee3-4665-abad-2977c25f68ba\") " pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" Nov 02 01:25:53.893769 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:53.893625 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7085c0ab-7ee3-4665-abad-2977c25f68ba-home\") pod \"premium-w5-123986-kserve-6d567889d6-fc7cn\" (UID: \"7085c0ab-7ee3-4665-abad-2977c25f68ba\") " pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" Nov 02 01:25:53.893769 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:53.893736 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7085c0ab-7ee3-4665-abad-2977c25f68ba-tmp-dir\") pod \"premium-w5-123986-kserve-6d567889d6-fc7cn\" (UID: \"7085c0ab-7ee3-4665-abad-2977c25f68ba\") " pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" Nov 02 01:25:53.893865 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:53.893823 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7085c0ab-7ee3-4665-abad-2977c25f68ba-dshm\") pod \"premium-w5-123986-kserve-6d567889d6-fc7cn\" (UID: \"7085c0ab-7ee3-4665-abad-2977c25f68ba\") " pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" Nov 02 01:25:53.893897 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:53.893882 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7085c0ab-7ee3-4665-abad-2977c25f68ba-tls-certs\") pod \"premium-w5-123986-kserve-6d567889d6-fc7cn\" (UID: \"7085c0ab-7ee3-4665-abad-2977c25f68ba\") " pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" Nov 02 01:25:53.893946 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:53.893934 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bx4t\" (UniqueName: \"kubernetes.io/projected/7085c0ab-7ee3-4665-abad-2977c25f68ba-kube-api-access-5bx4t\") pod \"premium-w5-123986-kserve-6d567889d6-fc7cn\" (UID: \"7085c0ab-7ee3-4665-abad-2977c25f68ba\") " pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" Nov 02 01:25:53.995284 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:53.995243 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7085c0ab-7ee3-4665-abad-2977c25f68ba-model-cache\") pod \"premium-w5-123986-kserve-6d567889d6-fc7cn\" (UID: \"7085c0ab-7ee3-4665-abad-2977c25f68ba\") " pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" Nov 02 01:25:53.995532 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:53.995340 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7085c0ab-7ee3-4665-abad-2977c25f68ba-home\") pod \"premium-w5-123986-kserve-6d567889d6-fc7cn\" (UID: \"7085c0ab-7ee3-4665-abad-2977c25f68ba\") " pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" Nov 02 01:25:53.995532 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:53.995401 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7085c0ab-7ee3-4665-abad-2977c25f68ba-tmp-dir\") pod \"premium-w5-123986-kserve-6d567889d6-fc7cn\" (UID: \"7085c0ab-7ee3-4665-abad-2977c25f68ba\") " pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" Nov 02 01:25:53.995532 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:53.995433 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7085c0ab-7ee3-4665-abad-2977c25f68ba-dshm\") pod \"premium-w5-123986-kserve-6d567889d6-fc7cn\" (UID: \"7085c0ab-7ee3-4665-abad-2977c25f68ba\") " pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" Nov 02 01:25:53.995532 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:53.995499 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7085c0ab-7ee3-4665-abad-2977c25f68ba-tls-certs\") pod \"premium-w5-123986-kserve-6d567889d6-fc7cn\" (UID: \"7085c0ab-7ee3-4665-abad-2977c25f68ba\") " pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" Nov 02 01:25:53.995532 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:53.995521 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bx4t\" (UniqueName: \"kubernetes.io/projected/7085c0ab-7ee3-4665-abad-2977c25f68ba-kube-api-access-5bx4t\") pod \"premium-w5-123986-kserve-6d567889d6-fc7cn\" (UID: \"7085c0ab-7ee3-4665-abad-2977c25f68ba\") " pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" Nov 02 01:25:53.995799 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:53.995754 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7085c0ab-7ee3-4665-abad-2977c25f68ba-home\") pod \"premium-w5-123986-kserve-6d567889d6-fc7cn\" (UID: \"7085c0ab-7ee3-4665-abad-2977c25f68ba\") " pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" Nov 02 01:25:53.995860 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:53.995825 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7085c0ab-7ee3-4665-abad-2977c25f68ba-tmp-dir\") pod \"premium-w5-123986-kserve-6d567889d6-fc7cn\" (UID: \"7085c0ab-7ee3-4665-abad-2977c25f68ba\") " pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" Nov 02 01:25:53.995996 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:53.995975 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7085c0ab-7ee3-4665-abad-2977c25f68ba-model-cache\") pod \"premium-w5-123986-kserve-6d567889d6-fc7cn\" (UID: \"7085c0ab-7ee3-4665-abad-2977c25f68ba\") " pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" Nov 02 01:25:53.997927 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:53.997894 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7085c0ab-7ee3-4665-abad-2977c25f68ba-dshm\") pod \"premium-w5-123986-kserve-6d567889d6-fc7cn\" (UID: \"7085c0ab-7ee3-4665-abad-2977c25f68ba\") " pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" Nov 02 01:25:53.998231 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:53.998209 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7085c0ab-7ee3-4665-abad-2977c25f68ba-tls-certs\") pod \"premium-w5-123986-kserve-6d567889d6-fc7cn\" (UID: \"7085c0ab-7ee3-4665-abad-2977c25f68ba\") " pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" Nov 02 01:25:54.007533 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:54.007509 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bx4t\" (UniqueName: \"kubernetes.io/projected/7085c0ab-7ee3-4665-abad-2977c25f68ba-kube-api-access-5bx4t\") pod \"premium-w5-123986-kserve-6d567889d6-fc7cn\" (UID: \"7085c0ab-7ee3-4665-abad-2977c25f68ba\") " pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" Nov 02 01:25:54.103904 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:54.103817 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" Nov 02 01:25:54.381313 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:54.381288 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn"] Nov 02 01:25:54.383465 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:25:54.383433 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7085c0ab_7ee3_4665_abad_2977c25f68ba.slice/crio-5ef37606d1bbc9aa2d4505b6487f53dc4f28893ee0039634d3efb2a4a5ff4e79 WatchSource:0}: Error finding container 5ef37606d1bbc9aa2d4505b6487f53dc4f28893ee0039634d3efb2a4a5ff4e79: Status 404 returned error can't find the container with id 5ef37606d1bbc9aa2d4505b6487f53dc4f28893ee0039634d3efb2a4a5ff4e79 Nov 02 01:25:54.621055 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:54.621010 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" event={"ID":"7085c0ab-7ee3-4665-abad-2977c25f68ba","Type":"ContainerStarted","Data":"327e8fa337e2c1f15185a409ce92ec7095017b734e922c2f41d63467bf181484"} Nov 02 01:25:54.621223 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:54.621068 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" event={"ID":"7085c0ab-7ee3-4665-abad-2977c25f68ba","Type":"ContainerStarted","Data":"5ef37606d1bbc9aa2d4505b6487f53dc4f28893ee0039634d3efb2a4a5ff4e79"} Nov 02 01:25:54.643851 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:54.643738 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" podStartSLOduration=1.6437250570000002 podStartE2EDuration="1.643725057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:25:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:25:54.641919833 +0000 UTC m=+659.262925047" watchObservedRunningTime="2026-11-02 01:25:54.643725057 +0000 UTC m=+659.264730268" Nov 02 01:25:55.104573 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:55.104463 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" Nov 02 01:25:57.087544 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:57.087497 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" Nov 02 01:25:57.088025 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:57.087737 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs"] Nov 02 01:25:57.092652 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:57.092628 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" Nov 02 01:25:57.097834 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:57.097811 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-47dd19\"/\"openshift-service-ca.crt\"" Nov 02 01:25:57.098016 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:57.098001 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-47dd19\"/\"kube-root-ca.crt\"" Nov 02 01:25:57.098081 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:57.098030 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-47dd19\"/\"default-dockercfg-4qqtf\"" Nov 02 01:25:57.098081 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:57.098048 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-47dd19\"/\"premium-w2-47dd19-kserve-self-signed-certs\"" Nov 02 01:25:57.103076 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:57.103042 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs"] Nov 02 01:25:57.104831 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:57.104807 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" Nov 02 01:25:57.235006 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:57.234962 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6-tmp-dir\") pod \"premium-w2-47dd19-kserve-68464dff57-wcgxs\" (UID: \"c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6\") " pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" Nov 02 01:25:57.236050 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:57.236023 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz4fv\" (UniqueName: \"kubernetes.io/projected/c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6-kube-api-access-vz4fv\") pod \"premium-w2-47dd19-kserve-68464dff57-wcgxs\" (UID: \"c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6\") " pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" Nov 02 01:25:57.236359 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:57.236339 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6-model-cache\") pod \"premium-w2-47dd19-kserve-68464dff57-wcgxs\" (UID: \"c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6\") " pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" Nov 02 01:25:57.241420 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:57.236562 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6-dshm\") pod \"premium-w2-47dd19-kserve-68464dff57-wcgxs\" (UID: \"c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6\") " pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" Nov 02 01:25:57.241655 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:57.241631 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6-tls-certs\") pod \"premium-w2-47dd19-kserve-68464dff57-wcgxs\" (UID: \"c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6\") " pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" Nov 02 01:25:57.241755 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:57.241670 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6-home\") pod \"premium-w2-47dd19-kserve-68464dff57-wcgxs\" (UID: \"c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6\") " pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" Nov 02 01:25:57.342989 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:57.342903 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6-model-cache\") pod \"premium-w2-47dd19-kserve-68464dff57-wcgxs\" (UID: \"c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6\") " pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" Nov 02 01:25:57.342989 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:57.342953 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6-dshm\") pod \"premium-w2-47dd19-kserve-68464dff57-wcgxs\" (UID: \"c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6\") " pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" Nov 02 01:25:57.343234 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:57.342992 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6-tls-certs\") pod \"premium-w2-47dd19-kserve-68464dff57-wcgxs\" (UID: \"c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6\") " pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" Nov 02 01:25:57.343234 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:57.343007 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6-home\") pod \"premium-w2-47dd19-kserve-68464dff57-wcgxs\" (UID: \"c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6\") " pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" Nov 02 01:25:57.343234 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:57.343038 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6-tmp-dir\") pod \"premium-w2-47dd19-kserve-68464dff57-wcgxs\" (UID: \"c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6\") " pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" Nov 02 01:25:57.343234 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:57.343062 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz4fv\" (UniqueName: \"kubernetes.io/projected/c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6-kube-api-access-vz4fv\") pod \"premium-w2-47dd19-kserve-68464dff57-wcgxs\" (UID: \"c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6\") " pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" Nov 02 01:25:57.343507 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:57.343430 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6-model-cache\") pod \"premium-w2-47dd19-kserve-68464dff57-wcgxs\" (UID: \"c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6\") " pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" Nov 02 01:25:57.343507 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:57.343474 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6-home\") pod \"premium-w2-47dd19-kserve-68464dff57-wcgxs\" (UID: \"c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6\") " pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" Nov 02 01:25:57.343589 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:57.343531 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6-tmp-dir\") pod \"premium-w2-47dd19-kserve-68464dff57-wcgxs\" (UID: \"c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6\") " pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" Nov 02 01:25:57.345562 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:57.345536 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6-dshm\") pod \"premium-w2-47dd19-kserve-68464dff57-wcgxs\" (UID: \"c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6\") " pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" Nov 02 01:25:57.345755 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:57.345734 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6-tls-certs\") pod \"premium-w2-47dd19-kserve-68464dff57-wcgxs\" (UID: \"c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6\") " pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" Nov 02 01:25:57.366415 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:57.366388 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz4fv\" (UniqueName: \"kubernetes.io/projected/c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6-kube-api-access-vz4fv\") pod \"premium-w2-47dd19-kserve-68464dff57-wcgxs\" (UID: \"c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6\") " pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" Nov 02 01:25:57.408771 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:57.408727 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" Nov 02 01:25:57.648155 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:57.648126 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" Nov 02 01:25:57.688730 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:57.688698 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs"] Nov 02 01:25:57.691585 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:25:57.691552 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc5dfd8e4_df57_4cfb_8ffd_d44dbb0273b6.slice/crio-380c736113518d7386efb87845d883df672246590fe211f6f764d5ac4c919a83 WatchSource:0}: Error finding container 380c736113518d7386efb87845d883df672246590fe211f6f764d5ac4c919a83: Status 404 returned error can't find the container with id 380c736113518d7386efb87845d883df672246590fe211f6f764d5ac4c919a83 Nov 02 01:25:58.643969 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:58.643921 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" event={"ID":"c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6","Type":"ContainerStarted","Data":"5ad14d31ea4019e07ab69f7baf7c92eeb8d8e042e2f0fd5cf41634783065770a"} Nov 02 01:25:58.644429 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:58.643979 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" event={"ID":"c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6","Type":"ContainerStarted","Data":"380c736113518d7386efb87845d883df672246590fe211f6f764d5ac4c919a83"} Nov 02 01:25:59.409962 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:25:59.409919 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" Nov 02 01:26:01.596695 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:26:01.596621 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:26:04.104336 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:04.104294 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" Nov 02 01:26:04.118431 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:04.118404 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" Nov 02 01:26:04.143989 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:04.143938 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" podStartSLOduration=7.143924443 podStartE2EDuration="7.143924443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:25:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:25:58.666270385 +0000 UTC m=+663.287275597" watchObservedRunningTime="2026-11-02 01:26:04.143924443 +0000 UTC m=+668.764929653" Nov 02 01:26:04.693022 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:04.692992 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" Nov 02 01:26:07.409841 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:07.409725 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" Nov 02 01:26:07.423506 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:07.423475 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" Nov 02 01:26:07.719574 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:07.719495 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" Nov 02 01:26:12.597276 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:12.593976 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj"] Nov 02 01:26:12.598190 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:12.598172 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" Nov 02 01:26:12.608249 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:12.608205 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5687bc\"/\"premium-w1-5687bc-kserve-self-signed-certs\"" Nov 02 01:26:12.611988 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:12.611947 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/706df1bd-19cf-4ab9-9aaa-3c12c97ccf45-home\") pod \"premium-w1-5687bc-kserve-5b5d47dd98-m7vbj\" (UID: \"706df1bd-19cf-4ab9-9aaa-3c12c97ccf45\") " pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" Nov 02 01:26:12.612099 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:12.611990 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/706df1bd-19cf-4ab9-9aaa-3c12c97ccf45-dshm\") pod \"premium-w1-5687bc-kserve-5b5d47dd98-m7vbj\" (UID: \"706df1bd-19cf-4ab9-9aaa-3c12c97ccf45\") " pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" Nov 02 01:26:12.612099 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:12.612040 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/706df1bd-19cf-4ab9-9aaa-3c12c97ccf45-tls-certs\") pod \"premium-w1-5687bc-kserve-5b5d47dd98-m7vbj\" (UID: \"706df1bd-19cf-4ab9-9aaa-3c12c97ccf45\") " pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" Nov 02 01:26:12.612208 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:12.612143 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clpz2\" (UniqueName: \"kubernetes.io/projected/706df1bd-19cf-4ab9-9aaa-3c12c97ccf45-kube-api-access-clpz2\") pod \"premium-w1-5687bc-kserve-5b5d47dd98-m7vbj\" (UID: \"706df1bd-19cf-4ab9-9aaa-3c12c97ccf45\") " pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" Nov 02 01:26:12.612208 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:12.612177 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/706df1bd-19cf-4ab9-9aaa-3c12c97ccf45-model-cache\") pod \"premium-w1-5687bc-kserve-5b5d47dd98-m7vbj\" (UID: \"706df1bd-19cf-4ab9-9aaa-3c12c97ccf45\") " pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" Nov 02 01:26:12.612295 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:12.612230 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/706df1bd-19cf-4ab9-9aaa-3c12c97ccf45-tmp-dir\") pod \"premium-w1-5687bc-kserve-5b5d47dd98-m7vbj\" (UID: \"706df1bd-19cf-4ab9-9aaa-3c12c97ccf45\") " pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" Nov 02 01:26:12.613986 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:12.613708 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj"] Nov 02 01:26:12.713859 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:12.713241 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/706df1bd-19cf-4ab9-9aaa-3c12c97ccf45-tmp-dir\") pod \"premium-w1-5687bc-kserve-5b5d47dd98-m7vbj\" (UID: \"706df1bd-19cf-4ab9-9aaa-3c12c97ccf45\") " pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" Nov 02 01:26:12.713859 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:12.713416 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/706df1bd-19cf-4ab9-9aaa-3c12c97ccf45-home\") pod \"premium-w1-5687bc-kserve-5b5d47dd98-m7vbj\" (UID: \"706df1bd-19cf-4ab9-9aaa-3c12c97ccf45\") " pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" Nov 02 01:26:12.713859 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:12.713473 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/706df1bd-19cf-4ab9-9aaa-3c12c97ccf45-dshm\") pod \"premium-w1-5687bc-kserve-5b5d47dd98-m7vbj\" (UID: \"706df1bd-19cf-4ab9-9aaa-3c12c97ccf45\") " pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" Nov 02 01:26:12.713859 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:12.713522 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/706df1bd-19cf-4ab9-9aaa-3c12c97ccf45-tls-certs\") pod \"premium-w1-5687bc-kserve-5b5d47dd98-m7vbj\" (UID: \"706df1bd-19cf-4ab9-9aaa-3c12c97ccf45\") " pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" Nov 02 01:26:12.713859 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:12.713595 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clpz2\" (UniqueName: \"kubernetes.io/projected/706df1bd-19cf-4ab9-9aaa-3c12c97ccf45-kube-api-access-clpz2\") pod \"premium-w1-5687bc-kserve-5b5d47dd98-m7vbj\" (UID: \"706df1bd-19cf-4ab9-9aaa-3c12c97ccf45\") " pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" Nov 02 01:26:12.713859 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:12.713629 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/706df1bd-19cf-4ab9-9aaa-3c12c97ccf45-model-cache\") pod \"premium-w1-5687bc-kserve-5b5d47dd98-m7vbj\" (UID: \"706df1bd-19cf-4ab9-9aaa-3c12c97ccf45\") " pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" Nov 02 01:26:12.713859 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:12.713710 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/706df1bd-19cf-4ab9-9aaa-3c12c97ccf45-tmp-dir\") pod \"premium-w1-5687bc-kserve-5b5d47dd98-m7vbj\" (UID: \"706df1bd-19cf-4ab9-9aaa-3c12c97ccf45\") " pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" Nov 02 01:26:12.714392 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:12.714292 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/706df1bd-19cf-4ab9-9aaa-3c12c97ccf45-model-cache\") pod \"premium-w1-5687bc-kserve-5b5d47dd98-m7vbj\" (UID: \"706df1bd-19cf-4ab9-9aaa-3c12c97ccf45\") " pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" Nov 02 01:26:12.714392 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:12.714322 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/706df1bd-19cf-4ab9-9aaa-3c12c97ccf45-home\") pod \"premium-w1-5687bc-kserve-5b5d47dd98-m7vbj\" (UID: \"706df1bd-19cf-4ab9-9aaa-3c12c97ccf45\") " pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" Nov 02 01:26:12.716183 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:12.716161 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/706df1bd-19cf-4ab9-9aaa-3c12c97ccf45-dshm\") pod \"premium-w1-5687bc-kserve-5b5d47dd98-m7vbj\" (UID: \"706df1bd-19cf-4ab9-9aaa-3c12c97ccf45\") " pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" Nov 02 01:26:12.716914 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:12.716879 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/706df1bd-19cf-4ab9-9aaa-3c12c97ccf45-tls-certs\") pod \"premium-w1-5687bc-kserve-5b5d47dd98-m7vbj\" (UID: \"706df1bd-19cf-4ab9-9aaa-3c12c97ccf45\") " pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" Nov 02 01:26:12.727153 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:12.727091 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clpz2\" (UniqueName: \"kubernetes.io/projected/706df1bd-19cf-4ab9-9aaa-3c12c97ccf45-kube-api-access-clpz2\") pod \"premium-w1-5687bc-kserve-5b5d47dd98-m7vbj\" (UID: \"706df1bd-19cf-4ab9-9aaa-3c12c97ccf45\") " pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" Nov 02 01:26:12.920102 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:12.920062 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" Nov 02 01:26:13.200104 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:13.200076 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj"] Nov 02 01:26:13.201731 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:26:13.201693 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod706df1bd_19cf_4ab9_9aaa_3c12c97ccf45.slice/crio-70918b0a9d6f77b25ec2d3a9478580f935485c810429c95d51c18b20f9d9d06b WatchSource:0}: Error finding container 70918b0a9d6f77b25ec2d3a9478580f935485c810429c95d51c18b20f9d9d06b: Status 404 returned error can't find the container with id 70918b0a9d6f77b25ec2d3a9478580f935485c810429c95d51c18b20f9d9d06b Nov 02 01:26:13.736996 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:13.736950 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" event={"ID":"706df1bd-19cf-4ab9-9aaa-3c12c97ccf45","Type":"ContainerStarted","Data":"01e63d12b58e032cd37fc5a34502a745c82a0b55ccec096b44dc22c39d2ad327"} Nov 02 01:26:13.737433 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:13.737002 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" event={"ID":"706df1bd-19cf-4ab9-9aaa-3c12c97ccf45","Type":"ContainerStarted","Data":"70918b0a9d6f77b25ec2d3a9478580f935485c810429c95d51c18b20f9d9d06b"} Nov 02 01:26:13.759449 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:13.759405 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" podStartSLOduration=1.759391505 podStartE2EDuration="1.759391505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:26:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:26:13.756259916 +0000 UTC m=+678.377265167" watchObservedRunningTime="2026-11-02 01:26:13.759391505 +0000 UTC m=+678.380396710" Nov 02 01:26:13.920843 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:13.920812 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" Nov 02 01:26:22.920539 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:22.920503 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" Nov 02 01:26:22.933846 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:22.933822 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" Nov 02 01:26:23.802602 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:23.802575 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" Nov 02 01:26:31.581981 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:26:31.581936 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:26:38.698997 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:38.698960 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm"] Nov 02 01:26:38.704067 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:38.704047 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm" Nov 02 01:26:38.708145 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:38.708128 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-b9d816-openshift-default-dockercfg-hvdvt\"" Nov 02 01:26:38.712197 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:38.712170 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/104f9b56-ef05-4c25-a485-4bbb6bac1ed0-istio-podinfo\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm\" (UID: \"104f9b56-ef05-4c25-a485-4bbb6bac1ed0\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm" Nov 02 01:26:38.712197 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:38.712206 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/104f9b56-ef05-4c25-a485-4bbb6bac1ed0-credential-socket\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm\" (UID: \"104f9b56-ef05-4c25-a485-4bbb6bac1ed0\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm" Nov 02 01:26:38.712417 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:38.712274 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/104f9b56-ef05-4c25-a485-4bbb6bac1ed0-istio-data\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm\" (UID: \"104f9b56-ef05-4c25-a485-4bbb6bac1ed0\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm" Nov 02 01:26:38.712417 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:38.712315 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/104f9b56-ef05-4c25-a485-4bbb6bac1ed0-workload-certs\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm\" (UID: \"104f9b56-ef05-4c25-a485-4bbb6bac1ed0\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm" Nov 02 01:26:38.712860 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:38.712421 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/104f9b56-ef05-4c25-a485-4bbb6bac1ed0-workload-socket\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm\" (UID: \"104f9b56-ef05-4c25-a485-4bbb6bac1ed0\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm" Nov 02 01:26:38.712860 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:38.712459 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq7bp\" (UniqueName: \"kubernetes.io/projected/104f9b56-ef05-4c25-a485-4bbb6bac1ed0-kube-api-access-kq7bp\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm\" (UID: \"104f9b56-ef05-4c25-a485-4bbb6bac1ed0\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm" Nov 02 01:26:38.712860 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:38.712493 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/104f9b56-ef05-4c25-a485-4bbb6bac1ed0-istiod-ca-cert\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm\" (UID: \"104f9b56-ef05-4c25-a485-4bbb6bac1ed0\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm" Nov 02 01:26:38.712860 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:38.712533 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/104f9b56-ef05-4c25-a485-4bbb6bac1ed0-istio-envoy\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm\" (UID: \"104f9b56-ef05-4c25-a485-4bbb6bac1ed0\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm" Nov 02 01:26:38.712860 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:38.712572 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/104f9b56-ef05-4c25-a485-4bbb6bac1ed0-istio-token\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm\" (UID: \"104f9b56-ef05-4c25-a485-4bbb6bac1ed0\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm" Nov 02 01:26:38.715605 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:38.715579 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm"] Nov 02 01:26:38.814140 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:38.814104 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/104f9b56-ef05-4c25-a485-4bbb6bac1ed0-workload-socket\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm\" (UID: \"104f9b56-ef05-4c25-a485-4bbb6bac1ed0\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm" Nov 02 01:26:38.814323 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:38.814145 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq7bp\" (UniqueName: \"kubernetes.io/projected/104f9b56-ef05-4c25-a485-4bbb6bac1ed0-kube-api-access-kq7bp\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm\" (UID: \"104f9b56-ef05-4c25-a485-4bbb6bac1ed0\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm" Nov 02 01:26:38.814323 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:38.814177 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/104f9b56-ef05-4c25-a485-4bbb6bac1ed0-istiod-ca-cert\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm\" (UID: \"104f9b56-ef05-4c25-a485-4bbb6bac1ed0\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm" Nov 02 01:26:38.814323 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:38.814211 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/104f9b56-ef05-4c25-a485-4bbb6bac1ed0-istio-envoy\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm\" (UID: \"104f9b56-ef05-4c25-a485-4bbb6bac1ed0\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm" Nov 02 01:26:38.814323 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:38.814247 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/104f9b56-ef05-4c25-a485-4bbb6bac1ed0-istio-token\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm\" (UID: \"104f9b56-ef05-4c25-a485-4bbb6bac1ed0\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm" Nov 02 01:26:38.814323 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:38.814287 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/104f9b56-ef05-4c25-a485-4bbb6bac1ed0-istio-podinfo\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm\" (UID: \"104f9b56-ef05-4c25-a485-4bbb6bac1ed0\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm" Nov 02 01:26:38.814323 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:38.814314 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/104f9b56-ef05-4c25-a485-4bbb6bac1ed0-credential-socket\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm\" (UID: \"104f9b56-ef05-4c25-a485-4bbb6bac1ed0\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm" Nov 02 01:26:38.814659 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:38.814408 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/104f9b56-ef05-4c25-a485-4bbb6bac1ed0-istio-data\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm\" (UID: \"104f9b56-ef05-4c25-a485-4bbb6bac1ed0\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm" Nov 02 01:26:38.814659 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:38.814446 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/104f9b56-ef05-4c25-a485-4bbb6bac1ed0-workload-certs\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm\" (UID: \"104f9b56-ef05-4c25-a485-4bbb6bac1ed0\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm" Nov 02 01:26:38.814659 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:38.814577 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/104f9b56-ef05-4c25-a485-4bbb6bac1ed0-workload-socket\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm\" (UID: \"104f9b56-ef05-4c25-a485-4bbb6bac1ed0\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm" Nov 02 01:26:38.814845 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:38.814822 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/104f9b56-ef05-4c25-a485-4bbb6bac1ed0-credential-socket\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm\" (UID: \"104f9b56-ef05-4c25-a485-4bbb6bac1ed0\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm" Nov 02 01:26:38.814845 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:38.814837 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/104f9b56-ef05-4c25-a485-4bbb6bac1ed0-workload-certs\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm\" (UID: \"104f9b56-ef05-4c25-a485-4bbb6bac1ed0\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm" Nov 02 01:26:38.814928 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:38.814905 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/104f9b56-ef05-4c25-a485-4bbb6bac1ed0-istio-data\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm\" (UID: \"104f9b56-ef05-4c25-a485-4bbb6bac1ed0\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm" Nov 02 01:26:38.815204 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:38.815181 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/104f9b56-ef05-4c25-a485-4bbb6bac1ed0-istiod-ca-cert\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm\" (UID: \"104f9b56-ef05-4c25-a485-4bbb6bac1ed0\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm" Nov 02 01:26:38.816720 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:38.816697 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/104f9b56-ef05-4c25-a485-4bbb6bac1ed0-istio-envoy\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm\" (UID: \"104f9b56-ef05-4c25-a485-4bbb6bac1ed0\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm" Nov 02 01:26:38.816969 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:38.816948 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/104f9b56-ef05-4c25-a485-4bbb6bac1ed0-istio-podinfo\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm\" (UID: \"104f9b56-ef05-4c25-a485-4bbb6bac1ed0\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm" Nov 02 01:26:38.826960 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:38.826931 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/104f9b56-ef05-4c25-a485-4bbb6bac1ed0-istio-token\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm\" (UID: \"104f9b56-ef05-4c25-a485-4bbb6bac1ed0\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm" Nov 02 01:26:38.827101 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:38.826993 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq7bp\" (UniqueName: \"kubernetes.io/projected/104f9b56-ef05-4c25-a485-4bbb6bac1ed0-kube-api-access-kq7bp\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm\" (UID: \"104f9b56-ef05-4c25-a485-4bbb6bac1ed0\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm" Nov 02 01:26:38.964222 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:38.964100 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8"] Nov 02 01:26:38.969189 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:38.969160 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8" Nov 02 01:26:38.977211 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:38.977178 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-b6ec07-openshift-default-dockercfg-9l7xh\"" Nov 02 01:26:38.988027 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:38.987991 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8"] Nov 02 01:26:39.015890 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:39.015859 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af735a61-695b-4a66-a7cb-d0df06b3b58b-istio-envoy\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8\" (UID: \"af735a61-695b-4a66-a7cb-d0df06b3b58b\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8" Nov 02 01:26:39.016096 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:39.015908 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf5cg\" (UniqueName: \"kubernetes.io/projected/af735a61-695b-4a66-a7cb-d0df06b3b58b-kube-api-access-wf5cg\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8\" (UID: \"af735a61-695b-4a66-a7cb-d0df06b3b58b\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8" Nov 02 01:26:39.016096 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:39.015970 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af735a61-695b-4a66-a7cb-d0df06b3b58b-workload-certs\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8\" (UID: \"af735a61-695b-4a66-a7cb-d0df06b3b58b\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8" Nov 02 01:26:39.016096 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:39.016018 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af735a61-695b-4a66-a7cb-d0df06b3b58b-istiod-ca-cert\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8\" (UID: \"af735a61-695b-4a66-a7cb-d0df06b3b58b\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8" Nov 02 01:26:39.016096 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:39.016088 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af735a61-695b-4a66-a7cb-d0df06b3b58b-istio-token\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8\" (UID: \"af735a61-695b-4a66-a7cb-d0df06b3b58b\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8" Nov 02 01:26:39.016321 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:39.016114 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af735a61-695b-4a66-a7cb-d0df06b3b58b-istio-data\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8\" (UID: \"af735a61-695b-4a66-a7cb-d0df06b3b58b\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8" Nov 02 01:26:39.016321 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:39.016144 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af735a61-695b-4a66-a7cb-d0df06b3b58b-credential-socket\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8\" (UID: \"af735a61-695b-4a66-a7cb-d0df06b3b58b\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8" Nov 02 01:26:39.016321 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:39.016173 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af735a61-695b-4a66-a7cb-d0df06b3b58b-workload-socket\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8\" (UID: \"af735a61-695b-4a66-a7cb-d0df06b3b58b\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8" Nov 02 01:26:39.016321 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:39.016205 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af735a61-695b-4a66-a7cb-d0df06b3b58b-istio-podinfo\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8\" (UID: \"af735a61-695b-4a66-a7cb-d0df06b3b58b\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8" Nov 02 01:26:39.020485 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:39.020458 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm" Nov 02 01:26:39.118191 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:39.118153 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af735a61-695b-4a66-a7cb-d0df06b3b58b-istio-token\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8\" (UID: \"af735a61-695b-4a66-a7cb-d0df06b3b58b\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8" Nov 02 01:26:39.124420 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:39.118208 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af735a61-695b-4a66-a7cb-d0df06b3b58b-istio-data\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8\" (UID: \"af735a61-695b-4a66-a7cb-d0df06b3b58b\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8" Nov 02 01:26:39.124420 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:39.118244 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af735a61-695b-4a66-a7cb-d0df06b3b58b-credential-socket\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8\" (UID: \"af735a61-695b-4a66-a7cb-d0df06b3b58b\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8" Nov 02 01:26:39.124420 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:39.118590 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af735a61-695b-4a66-a7cb-d0df06b3b58b-workload-socket\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8\" (UID: \"af735a61-695b-4a66-a7cb-d0df06b3b58b\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8" Nov 02 01:26:39.124420 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:39.118632 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af735a61-695b-4a66-a7cb-d0df06b3b58b-istio-podinfo\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8\" (UID: \"af735a61-695b-4a66-a7cb-d0df06b3b58b\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8" Nov 02 01:26:39.124420 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:39.118699 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af735a61-695b-4a66-a7cb-d0df06b3b58b-istio-envoy\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8\" (UID: \"af735a61-695b-4a66-a7cb-d0df06b3b58b\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8" Nov 02 01:26:39.124420 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:39.118738 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf5cg\" (UniqueName: \"kubernetes.io/projected/af735a61-695b-4a66-a7cb-d0df06b3b58b-kube-api-access-wf5cg\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8\" (UID: \"af735a61-695b-4a66-a7cb-d0df06b3b58b\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8" Nov 02 01:26:39.124420 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:39.118818 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af735a61-695b-4a66-a7cb-d0df06b3b58b-workload-certs\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8\" (UID: \"af735a61-695b-4a66-a7cb-d0df06b3b58b\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8" Nov 02 01:26:39.124420 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:39.118878 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af735a61-695b-4a66-a7cb-d0df06b3b58b-istiod-ca-cert\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8\" (UID: \"af735a61-695b-4a66-a7cb-d0df06b3b58b\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8" Nov 02 01:26:39.126252 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:39.126218 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af735a61-695b-4a66-a7cb-d0df06b3b58b-credential-socket\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8\" (UID: \"af735a61-695b-4a66-a7cb-d0df06b3b58b\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8" Nov 02 01:26:39.126687 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:39.126660 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af735a61-695b-4a66-a7cb-d0df06b3b58b-istiod-ca-cert\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8\" (UID: \"af735a61-695b-4a66-a7cb-d0df06b3b58b\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8" Nov 02 01:26:39.127231 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:39.127185 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af735a61-695b-4a66-a7cb-d0df06b3b58b-workload-certs\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8\" (UID: \"af735a61-695b-4a66-a7cb-d0df06b3b58b\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8" Nov 02 01:26:39.127628 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:39.127582 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af735a61-695b-4a66-a7cb-d0df06b3b58b-istio-data\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8\" (UID: \"af735a61-695b-4a66-a7cb-d0df06b3b58b\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8" Nov 02 01:26:39.128201 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:39.127929 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af735a61-695b-4a66-a7cb-d0df06b3b58b-workload-socket\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8\" (UID: \"af735a61-695b-4a66-a7cb-d0df06b3b58b\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8" Nov 02 01:26:39.130798 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:39.130223 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af735a61-695b-4a66-a7cb-d0df06b3b58b-istio-envoy\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8\" (UID: \"af735a61-695b-4a66-a7cb-d0df06b3b58b\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8" Nov 02 01:26:39.130798 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:39.130743 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af735a61-695b-4a66-a7cb-d0df06b3b58b-istio-podinfo\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8\" (UID: \"af735a61-695b-4a66-a7cb-d0df06b3b58b\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8" Nov 02 01:26:39.162524 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:39.162491 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf5cg\" (UniqueName: \"kubernetes.io/projected/af735a61-695b-4a66-a7cb-d0df06b3b58b-kube-api-access-wf5cg\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8\" (UID: \"af735a61-695b-4a66-a7cb-d0df06b3b58b\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8" Nov 02 01:26:39.163102 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:39.163080 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af735a61-695b-4a66-a7cb-d0df06b3b58b-istio-token\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8\" (UID: \"af735a61-695b-4a66-a7cb-d0df06b3b58b\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8" Nov 02 01:26:39.285449 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:39.284927 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8" Nov 02 01:26:39.318133 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:39.317555 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm"] Nov 02 01:26:39.324716 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:26:39.324672 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod104f9b56_ef05_4c25_a485_4bbb6bac1ed0.slice/crio-2008039fd6c3bfcdae4c5d052caf004a6a3a093c3ba7dc4008cc83105209f90e WatchSource:0}: Error finding container 2008039fd6c3bfcdae4c5d052caf004a6a3a093c3ba7dc4008cc83105209f90e: Status 404 returned error can't find the container with id 2008039fd6c3bfcdae4c5d052caf004a6a3a093c3ba7dc4008cc83105209f90e Nov 02 01:26:39.330420 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:39.329818 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:26:39.330420 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:39.329907 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:26:39.330420 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:39.329984 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:26:39.585707 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:39.585675 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8"] Nov 02 01:26:39.588904 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:26:39.587883 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf735a61_695b_4a66_a7cb_d0df06b3b58b.slice/crio-b4d7f9852bb0b2fb7dc983f0952de09e4b4b4d71d4aff7f2ea71ea4543c65936 WatchSource:0}: Error finding container b4d7f9852bb0b2fb7dc983f0952de09e4b4b4d71d4aff7f2ea71ea4543c65936: Status 404 returned error can't find the container with id b4d7f9852bb0b2fb7dc983f0952de09e4b4b4d71d4aff7f2ea71ea4543c65936 Nov 02 01:26:39.591195 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:39.591161 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:26:39.591318 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:39.591234 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:26:39.591318 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:39.591264 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:26:39.894400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:39.889671 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm" event={"ID":"104f9b56-ef05-4c25-a485-4bbb6bac1ed0","Type":"ContainerStarted","Data":"421335e5adc15f5080e7e69849a5454438b4f77a765620f94dc1b0b5e0a3fa6a"} Nov 02 01:26:39.894400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:39.889764 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm" event={"ID":"104f9b56-ef05-4c25-a485-4bbb6bac1ed0","Type":"ContainerStarted","Data":"2008039fd6c3bfcdae4c5d052caf004a6a3a093c3ba7dc4008cc83105209f90e"} Nov 02 01:26:39.894400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:39.892938 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8" event={"ID":"af735a61-695b-4a66-a7cb-d0df06b3b58b","Type":"ContainerStarted","Data":"40e8a42407d0d5421dd8d43594a03c7f6cdcf21528adc613f5c28d4bf54a2c65"} Nov 02 01:26:39.894400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:39.892993 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8" event={"ID":"af735a61-695b-4a66-a7cb-d0df06b3b58b","Type":"ContainerStarted","Data":"b4d7f9852bb0b2fb7dc983f0952de09e4b4b4d71d4aff7f2ea71ea4543c65936"} Nov 02 01:26:39.917286 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:39.917224 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm" podStartSLOduration=1.917204087 podStartE2EDuration="1.917204087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:26:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:26:39.915753772 +0000 UTC m=+704.536758984" watchObservedRunningTime="2026-11-02 01:26:39.917204087 +0000 UTC m=+704.538209300" Nov 02 01:26:39.945224 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:39.945155 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8" podStartSLOduration=1.9451339380000001 podStartE2EDuration="1.945133938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:26:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:26:39.941865529 +0000 UTC m=+704.562870751" watchObservedRunningTime="2026-11-02 01:26:39.945133938 +0000 UTC m=+704.566139151" Nov 02 01:26:40.023484 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:40.023450 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm" Nov 02 01:26:40.027513 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:40.027477 4427 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 02 01:26:40.027727 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:40.027697 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm" podUID="104f9b56-ef05-4c25-a485-4bbb6bac1ed0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 02 01:26:40.286464 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:40.285762 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8" Nov 02 01:26:40.302399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:40.291504 4427 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Nov 02 01:26:40.302399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:40.291566 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8" podUID="af735a61-695b-4a66-a7cb-d0df06b3b58b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Nov 02 01:26:41.022181 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:41.022134 4427 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 02 01:26:41.022615 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:41.022203 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm" podUID="104f9b56-ef05-4c25-a485-4bbb6bac1ed0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 02 01:26:41.290494 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:41.289896 4427 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Nov 02 01:26:41.290494 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:41.289976 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8" podUID="af735a61-695b-4a66-a7cb-d0df06b3b58b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Nov 02 01:26:42.023395 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:42.022190 4427 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 02 01:26:42.023395 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:42.022271 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm" podUID="104f9b56-ef05-4c25-a485-4bbb6bac1ed0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 02 01:26:42.288457 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:42.287865 4427 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Nov 02 01:26:42.288457 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:42.287940 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8" podUID="af735a61-695b-4a66-a7cb-d0df06b3b58b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Nov 02 01:26:43.021286 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:43.021245 4427 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 02 01:26:43.021530 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:43.021298 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm" podUID="104f9b56-ef05-4c25-a485-4bbb6bac1ed0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 02 01:26:43.294859 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:43.294762 4427 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Nov 02 01:26:43.295471 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:43.295440 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8" podUID="af735a61-695b-4a66-a7cb-d0df06b3b58b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Nov 02 01:26:44.021462 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:44.021425 4427 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 02 01:26:44.021724 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:44.021691 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm" podUID="104f9b56-ef05-4c25-a485-4bbb6bac1ed0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 02 01:26:44.288456 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:44.287726 4427 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Nov 02 01:26:44.288456 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:44.287784 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8" podUID="af735a61-695b-4a66-a7cb-d0df06b3b58b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Nov 02 01:26:45.022313 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:45.022272 4427 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 02 01:26:45.022743 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:45.022342 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm" podUID="104f9b56-ef05-4c25-a485-4bbb6bac1ed0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 02 01:26:45.287324 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:45.286806 4427 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Nov 02 01:26:45.287324 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:45.286869 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8" podUID="af735a61-695b-4a66-a7cb-d0df06b3b58b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Nov 02 01:26:46.202215 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:46.202180 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm" Nov 02 01:26:46.203507 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:46.203481 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm" Nov 02 01:26:46.204583 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:46.204560 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm" Nov 02 01:26:46.540817 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:46.540721 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8" Nov 02 01:26:46.541224 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:46.541200 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8" Nov 02 01:26:46.542330 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:46.542295 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8" Nov 02 01:26:53.799034 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:53.799002 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2"] Nov 02 01:26:53.804698 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:53.804680 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2" Nov 02 01:26:53.811188 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:53.811149 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-5687bc-openshift-default-dockercfg-mw8fh\"" Nov 02 01:26:53.815332 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:53.815306 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2"] Nov 02 01:26:53.938868 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:53.938827 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bcaee75-ac15-4b13-8f60-401ee60464ec-istiod-ca-cert\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2\" (UID: \"2bcaee75-ac15-4b13-8f60-401ee60464ec\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2" Nov 02 01:26:53.939135 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:53.939116 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bcaee75-ac15-4b13-8f60-401ee60464ec-istio-podinfo\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2\" (UID: \"2bcaee75-ac15-4b13-8f60-401ee60464ec\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2" Nov 02 01:26:53.939252 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:53.939235 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kfwl\" (UniqueName: \"kubernetes.io/projected/2bcaee75-ac15-4b13-8f60-401ee60464ec-kube-api-access-6kfwl\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2\" (UID: \"2bcaee75-ac15-4b13-8f60-401ee60464ec\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2" Nov 02 01:26:53.939395 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:53.939356 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bcaee75-ac15-4b13-8f60-401ee60464ec-istio-data\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2\" (UID: \"2bcaee75-ac15-4b13-8f60-401ee60464ec\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2" Nov 02 01:26:53.939585 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:53.939569 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bcaee75-ac15-4b13-8f60-401ee60464ec-istio-envoy\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2\" (UID: \"2bcaee75-ac15-4b13-8f60-401ee60464ec\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2" Nov 02 01:26:53.939690 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:53.939677 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bcaee75-ac15-4b13-8f60-401ee60464ec-workload-socket\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2\" (UID: \"2bcaee75-ac15-4b13-8f60-401ee60464ec\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2" Nov 02 01:26:53.939812 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:53.939799 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bcaee75-ac15-4b13-8f60-401ee60464ec-workload-certs\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2\" (UID: \"2bcaee75-ac15-4b13-8f60-401ee60464ec\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2" Nov 02 01:26:53.939924 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:53.939912 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bcaee75-ac15-4b13-8f60-401ee60464ec-credential-socket\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2\" (UID: \"2bcaee75-ac15-4b13-8f60-401ee60464ec\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2" Nov 02 01:26:53.940063 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:53.940048 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bcaee75-ac15-4b13-8f60-401ee60464ec-istio-token\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2\" (UID: \"2bcaee75-ac15-4b13-8f60-401ee60464ec\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2" Nov 02 01:26:54.041531 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:54.041495 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bcaee75-ac15-4b13-8f60-401ee60464ec-istio-envoy\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2\" (UID: \"2bcaee75-ac15-4b13-8f60-401ee60464ec\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2" Nov 02 01:26:54.041531 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:54.041532 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bcaee75-ac15-4b13-8f60-401ee60464ec-workload-socket\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2\" (UID: \"2bcaee75-ac15-4b13-8f60-401ee60464ec\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2" Nov 02 01:26:54.041782 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:54.041571 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bcaee75-ac15-4b13-8f60-401ee60464ec-workload-certs\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2\" (UID: \"2bcaee75-ac15-4b13-8f60-401ee60464ec\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2" Nov 02 01:26:54.041782 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:54.041616 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bcaee75-ac15-4b13-8f60-401ee60464ec-credential-socket\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2\" (UID: \"2bcaee75-ac15-4b13-8f60-401ee60464ec\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2" Nov 02 01:26:54.041782 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:54.041654 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bcaee75-ac15-4b13-8f60-401ee60464ec-istio-token\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2\" (UID: \"2bcaee75-ac15-4b13-8f60-401ee60464ec\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2" Nov 02 01:26:54.041782 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:54.041757 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bcaee75-ac15-4b13-8f60-401ee60464ec-istiod-ca-cert\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2\" (UID: \"2bcaee75-ac15-4b13-8f60-401ee60464ec\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2" Nov 02 01:26:54.041997 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:54.041785 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bcaee75-ac15-4b13-8f60-401ee60464ec-istio-podinfo\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2\" (UID: \"2bcaee75-ac15-4b13-8f60-401ee60464ec\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2" Nov 02 01:26:54.041997 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:54.041810 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kfwl\" (UniqueName: \"kubernetes.io/projected/2bcaee75-ac15-4b13-8f60-401ee60464ec-kube-api-access-6kfwl\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2\" (UID: \"2bcaee75-ac15-4b13-8f60-401ee60464ec\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2" Nov 02 01:26:54.041997 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:54.041841 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bcaee75-ac15-4b13-8f60-401ee60464ec-istio-data\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2\" (UID: \"2bcaee75-ac15-4b13-8f60-401ee60464ec\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2" Nov 02 01:26:54.042151 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:54.042077 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bcaee75-ac15-4b13-8f60-401ee60464ec-workload-certs\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2\" (UID: \"2bcaee75-ac15-4b13-8f60-401ee60464ec\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2" Nov 02 01:26:54.042219 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:54.042196 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bcaee75-ac15-4b13-8f60-401ee60464ec-credential-socket\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2\" (UID: \"2bcaee75-ac15-4b13-8f60-401ee60464ec\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2" Nov 02 01:26:54.042258 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:54.042243 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bcaee75-ac15-4b13-8f60-401ee60464ec-istio-data\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2\" (UID: \"2bcaee75-ac15-4b13-8f60-401ee60464ec\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2" Nov 02 01:26:54.042531 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:54.042510 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bcaee75-ac15-4b13-8f60-401ee60464ec-istiod-ca-cert\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2\" (UID: \"2bcaee75-ac15-4b13-8f60-401ee60464ec\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2" Nov 02 01:26:54.042654 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:54.042632 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bcaee75-ac15-4b13-8f60-401ee60464ec-workload-socket\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2\" (UID: \"2bcaee75-ac15-4b13-8f60-401ee60464ec\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2" Nov 02 01:26:54.044168 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:54.044144 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bcaee75-ac15-4b13-8f60-401ee60464ec-istio-envoy\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2\" (UID: \"2bcaee75-ac15-4b13-8f60-401ee60464ec\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2" Nov 02 01:26:54.044668 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:54.044647 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bcaee75-ac15-4b13-8f60-401ee60464ec-istio-podinfo\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2\" (UID: \"2bcaee75-ac15-4b13-8f60-401ee60464ec\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2" Nov 02 01:26:54.053345 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:54.053252 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bcaee75-ac15-4b13-8f60-401ee60464ec-istio-token\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2\" (UID: \"2bcaee75-ac15-4b13-8f60-401ee60464ec\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2" Nov 02 01:26:54.053553 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:54.053534 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kfwl\" (UniqueName: \"kubernetes.io/projected/2bcaee75-ac15-4b13-8f60-401ee60464ec-kube-api-access-6kfwl\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2\" (UID: \"2bcaee75-ac15-4b13-8f60-401ee60464ec\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2" Nov 02 01:26:54.120855 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:54.120822 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2" Nov 02 01:26:54.214110 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:54.214073 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4"] Nov 02 01:26:54.220101 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:54.220072 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4" Nov 02 01:26:54.231167 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:54.226450 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-5ed42b-openshift-default-dockercfg-6p624\"" Nov 02 01:26:54.231167 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:54.230723 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4"] Nov 02 01:26:54.344637 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:54.344552 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb3258a9-adfb-46de-a0ab-7b4649948689-istio-data\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4\" (UID: \"eb3258a9-adfb-46de-a0ab-7b4649948689\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4" Nov 02 01:26:54.344637 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:54.344591 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb3258a9-adfb-46de-a0ab-7b4649948689-istiod-ca-cert\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4\" (UID: \"eb3258a9-adfb-46de-a0ab-7b4649948689\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4" Nov 02 01:26:54.344832 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:54.344766 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb3258a9-adfb-46de-a0ab-7b4649948689-credential-socket\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4\" (UID: \"eb3258a9-adfb-46de-a0ab-7b4649948689\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4" Nov 02 01:26:54.344889 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:54.344834 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb3258a9-adfb-46de-a0ab-7b4649948689-istio-token\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4\" (UID: \"eb3258a9-adfb-46de-a0ab-7b4649948689\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4" Nov 02 01:26:54.344943 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:54.344922 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb3258a9-adfb-46de-a0ab-7b4649948689-workload-socket\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4\" (UID: \"eb3258a9-adfb-46de-a0ab-7b4649948689\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4" Nov 02 01:26:54.345024 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:54.345005 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb3258a9-adfb-46de-a0ab-7b4649948689-istio-envoy\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4\" (UID: \"eb3258a9-adfb-46de-a0ab-7b4649948689\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4" Nov 02 01:26:54.345083 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:54.345037 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb3258a9-adfb-46de-a0ab-7b4649948689-istio-podinfo\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4\" (UID: \"eb3258a9-adfb-46de-a0ab-7b4649948689\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4" Nov 02 01:26:54.345142 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:54.345121 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb3258a9-adfb-46de-a0ab-7b4649948689-workload-certs\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4\" (UID: \"eb3258a9-adfb-46de-a0ab-7b4649948689\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4" Nov 02 01:26:54.345184 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:54.345173 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s672s\" (UniqueName: \"kubernetes.io/projected/eb3258a9-adfb-46de-a0ab-7b4649948689-kube-api-access-s672s\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4\" (UID: \"eb3258a9-adfb-46de-a0ab-7b4649948689\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4" Nov 02 01:26:54.403820 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:54.403792 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2"] Nov 02 01:26:54.406663 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:26:54.406631 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bcaee75_ac15_4b13_8f60_401ee60464ec.slice/crio-b42249434a59c6c6e7df6e7cff34b2aa31dcd7aed42f2b4b2b27de5ac6185e8f WatchSource:0}: Error finding container b42249434a59c6c6e7df6e7cff34b2aa31dcd7aed42f2b4b2b27de5ac6185e8f: Status 404 returned error can't find the container with id b42249434a59c6c6e7df6e7cff34b2aa31dcd7aed42f2b4b2b27de5ac6185e8f Nov 02 01:26:54.409144 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:54.409087 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:26:54.409250 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:54.409176 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:26:54.409250 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:54.409220 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:26:54.446619 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:54.446590 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb3258a9-adfb-46de-a0ab-7b4649948689-credential-socket\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4\" (UID: \"eb3258a9-adfb-46de-a0ab-7b4649948689\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4" Nov 02 01:26:54.446823 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:54.446655 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb3258a9-adfb-46de-a0ab-7b4649948689-istio-token\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4\" (UID: \"eb3258a9-adfb-46de-a0ab-7b4649948689\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4" Nov 02 01:26:54.446823 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:54.446675 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb3258a9-adfb-46de-a0ab-7b4649948689-workload-socket\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4\" (UID: \"eb3258a9-adfb-46de-a0ab-7b4649948689\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4" Nov 02 01:26:54.446823 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:54.446707 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb3258a9-adfb-46de-a0ab-7b4649948689-istio-envoy\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4\" (UID: \"eb3258a9-adfb-46de-a0ab-7b4649948689\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4" Nov 02 01:26:54.446823 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:54.446722 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb3258a9-adfb-46de-a0ab-7b4649948689-istio-podinfo\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4\" (UID: \"eb3258a9-adfb-46de-a0ab-7b4649948689\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4" Nov 02 01:26:54.446823 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:54.446762 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb3258a9-adfb-46de-a0ab-7b4649948689-workload-certs\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4\" (UID: \"eb3258a9-adfb-46de-a0ab-7b4649948689\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4" Nov 02 01:26:54.446823 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:54.446798 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s672s\" (UniqueName: \"kubernetes.io/projected/eb3258a9-adfb-46de-a0ab-7b4649948689-kube-api-access-s672s\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4\" (UID: \"eb3258a9-adfb-46de-a0ab-7b4649948689\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4" Nov 02 01:26:54.447169 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:54.446841 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb3258a9-adfb-46de-a0ab-7b4649948689-istio-data\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4\" (UID: \"eb3258a9-adfb-46de-a0ab-7b4649948689\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4" Nov 02 01:26:54.447169 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:54.446871 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb3258a9-adfb-46de-a0ab-7b4649948689-istiod-ca-cert\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4\" (UID: \"eb3258a9-adfb-46de-a0ab-7b4649948689\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4" Nov 02 01:26:54.447169 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:54.447053 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb3258a9-adfb-46de-a0ab-7b4649948689-credential-socket\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4\" (UID: \"eb3258a9-adfb-46de-a0ab-7b4649948689\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4" Nov 02 01:26:54.447169 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:54.447125 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb3258a9-adfb-46de-a0ab-7b4649948689-workload-socket\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4\" (UID: \"eb3258a9-adfb-46de-a0ab-7b4649948689\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4" Nov 02 01:26:54.447409 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:54.447325 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb3258a9-adfb-46de-a0ab-7b4649948689-istio-data\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4\" (UID: \"eb3258a9-adfb-46de-a0ab-7b4649948689\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4" Nov 02 01:26:54.447409 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:54.447326 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb3258a9-adfb-46de-a0ab-7b4649948689-workload-certs\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4\" (UID: \"eb3258a9-adfb-46de-a0ab-7b4649948689\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4" Nov 02 01:26:54.448268 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:54.448241 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb3258a9-adfb-46de-a0ab-7b4649948689-istiod-ca-cert\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4\" (UID: \"eb3258a9-adfb-46de-a0ab-7b4649948689\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4" Nov 02 01:26:54.449583 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:54.449544 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb3258a9-adfb-46de-a0ab-7b4649948689-istio-envoy\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4\" (UID: \"eb3258a9-adfb-46de-a0ab-7b4649948689\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4" Nov 02 01:26:54.449808 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:54.449788 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb3258a9-adfb-46de-a0ab-7b4649948689-istio-podinfo\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4\" (UID: \"eb3258a9-adfb-46de-a0ab-7b4649948689\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4" Nov 02 01:26:54.458824 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:54.458802 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb3258a9-adfb-46de-a0ab-7b4649948689-istio-token\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4\" (UID: \"eb3258a9-adfb-46de-a0ab-7b4649948689\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4" Nov 02 01:26:54.459099 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:54.459071 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s672s\" (UniqueName: \"kubernetes.io/projected/eb3258a9-adfb-46de-a0ab-7b4649948689-kube-api-access-s672s\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4\" (UID: \"eb3258a9-adfb-46de-a0ab-7b4649948689\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4" Nov 02 01:26:54.540426 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:54.539949 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4" Nov 02 01:26:54.842486 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:54.842442 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4"] Nov 02 01:26:54.846562 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:26:54.846525 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb3258a9_adfb_46de_a0ab_7b4649948689.slice/crio-3eef0c53616448963b113a190dbc03bea2dad6315d1fa9e9e09eba5bf68b3eaa WatchSource:0}: Error finding container 3eef0c53616448963b113a190dbc03bea2dad6315d1fa9e9e09eba5bf68b3eaa: Status 404 returned error can't find the container with id 3eef0c53616448963b113a190dbc03bea2dad6315d1fa9e9e09eba5bf68b3eaa Nov 02 01:26:54.849012 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:54.848982 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:26:54.849090 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:54.849047 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:26:54.849090 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:54.849074 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:26:55.012274 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:55.012238 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2" event={"ID":"2bcaee75-ac15-4b13-8f60-401ee60464ec","Type":"ContainerStarted","Data":"2e79b85cda3427b9b65f6c6554d1ef37b6df7d7b079e9b9f9fb4b7b705b61891"} Nov 02 01:26:55.012491 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:55.012297 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2" event={"ID":"2bcaee75-ac15-4b13-8f60-401ee60464ec","Type":"ContainerStarted","Data":"b42249434a59c6c6e7df6e7cff34b2aa31dcd7aed42f2b4b2b27de5ac6185e8f"} Nov 02 01:26:55.013994 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:55.013962 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4" event={"ID":"eb3258a9-adfb-46de-a0ab-7b4649948689","Type":"ContainerStarted","Data":"1b5572190e14ea57e5510ae221581192ac1496c471d71938270e59a7d347f04a"} Nov 02 01:26:55.014485 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:55.014018 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4" event={"ID":"eb3258a9-adfb-46de-a0ab-7b4649948689","Type":"ContainerStarted","Data":"3eef0c53616448963b113a190dbc03bea2dad6315d1fa9e9e09eba5bf68b3eaa"} Nov 02 01:26:55.037879 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:55.037823 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2" podStartSLOduration=2.037808062 podStartE2EDuration="2.037808062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:26:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:26:55.034332772 +0000 UTC m=+719.655337981" watchObservedRunningTime="2026-11-02 01:26:55.037808062 +0000 UTC m=+719.658813273" Nov 02 01:26:55.121655 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:55.121619 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2" Nov 02 01:26:55.123577 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:55.123544 4427 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Nov 02 01:26:55.123699 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:55.123609 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2" podUID="2bcaee75-ac15-4b13-8f60-401ee60464ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Nov 02 01:26:55.158935 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:55.158834 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4" podStartSLOduration=1.158814389 podStartE2EDuration="1.158814389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:26:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:26:55.070704699 +0000 UTC m=+719.691709912" watchObservedRunningTime="2026-11-02 01:26:55.158814389 +0000 UTC m=+719.779819603" Nov 02 01:26:55.180401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:55.178432 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3190c013-k92s4"] Nov 02 01:26:55.199890 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:55.197779 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3190c013-k92s4"] Nov 02 01:26:55.199890 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:55.197918 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3190c013-k92s4" Nov 02 01:26:55.203249 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:55.203220 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 01:26:55.205438 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:55.203318 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\"" Nov 02 01:26:55.256330 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:55.256292 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpwqs\" (UniqueName: \"kubernetes.io/projected/92b288ee-b9e7-494a-a48b-0452b09e9f03-kube-api-access-wpwqs\") pod \"maas-api-revoke-sub-3190c013-k92s4\" (UID: \"92b288ee-b9e7-494a-a48b-0452b09e9f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3190c013-k92s4" Nov 02 01:26:55.256330 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:55.256332 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92b288ee-b9e7-494a-a48b-0452b09e9f03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3190c013-k92s4\" (UID: \"92b288ee-b9e7-494a-a48b-0452b09e9f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3190c013-k92s4" Nov 02 01:26:55.256605 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:55.256519 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92b288ee-b9e7-494a-a48b-0452b09e9f03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3190c013-k92s4\" (UID: \"92b288ee-b9e7-494a-a48b-0452b09e9f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3190c013-k92s4" Nov 02 01:26:55.357464 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:55.357415 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpwqs\" (UniqueName: \"kubernetes.io/projected/92b288ee-b9e7-494a-a48b-0452b09e9f03-kube-api-access-wpwqs\") pod \"maas-api-revoke-sub-3190c013-k92s4\" (UID: \"92b288ee-b9e7-494a-a48b-0452b09e9f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3190c013-k92s4" Nov 02 01:26:55.357464 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:55.357452 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92b288ee-b9e7-494a-a48b-0452b09e9f03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3190c013-k92s4\" (UID: \"92b288ee-b9e7-494a-a48b-0452b09e9f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3190c013-k92s4" Nov 02 01:26:55.357788 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:55.357516 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92b288ee-b9e7-494a-a48b-0452b09e9f03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3190c013-k92s4\" (UID: \"92b288ee-b9e7-494a-a48b-0452b09e9f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3190c013-k92s4" Nov 02 01:26:55.358130 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:55.358105 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92b288ee-b9e7-494a-a48b-0452b09e9f03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3190c013-k92s4\" (UID: \"92b288ee-b9e7-494a-a48b-0452b09e9f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3190c013-k92s4" Nov 02 01:26:55.383400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:55.382429 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92b288ee-b9e7-494a-a48b-0452b09e9f03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3190c013-k92s4\" (UID: \"92b288ee-b9e7-494a-a48b-0452b09e9f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3190c013-k92s4" Nov 02 01:26:55.390670 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:55.390638 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpwqs\" (UniqueName: \"kubernetes.io/projected/92b288ee-b9e7-494a-a48b-0452b09e9f03-kube-api-access-wpwqs\") pod \"maas-api-revoke-sub-3190c013-k92s4\" (UID: \"92b288ee-b9e7-494a-a48b-0452b09e9f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3190c013-k92s4" Nov 02 01:26:55.524274 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:55.524195 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\"" Nov 02 01:26:55.532410 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:55.531457 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3190c013-k92s4" Nov 02 01:26:55.542212 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:55.541923 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4" Nov 02 01:26:55.543499 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:55.543474 4427 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 02 01:26:55.543604 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:55.543523 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4" podUID="eb3258a9-adfb-46de-a0ab-7b4649948689" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 02 01:26:55.908552 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:55.908312 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3190c013-k92s4"] Nov 02 01:26:55.908552 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:26:55.908522 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92b288ee_b9e7_494a_a48b_0452b09e9f03.slice/crio-94ee40b61b171b54af455d18edf1c0fd1d97dabcbe10f7fae4b0e0a5468a34cd WatchSource:0}: Error finding container 94ee40b61b171b54af455d18edf1c0fd1d97dabcbe10f7fae4b0e0a5468a34cd: Status 404 returned error can't find the container with id 94ee40b61b171b54af455d18edf1c0fd1d97dabcbe10f7fae4b0e0a5468a34cd Nov 02 01:26:56.022554 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:56.022518 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3190c013-k92s4" event={"ID":"92b288ee-b9e7-494a-a48b-0452b09e9f03","Type":"ContainerStarted","Data":"94ee40b61b171b54af455d18edf1c0fd1d97dabcbe10f7fae4b0e0a5468a34cd"} Nov 02 01:26:56.122603 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:56.122570 4427 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Nov 02 01:26:56.122754 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:56.122620 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2" podUID="2bcaee75-ac15-4b13-8f60-401ee60464ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Nov 02 01:26:56.542414 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:56.542350 4427 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 02 01:26:56.542591 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:56.542437 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4" podUID="eb3258a9-adfb-46de-a0ab-7b4649948689" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 02 01:26:57.121726 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:57.121402 4427 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Nov 02 01:26:57.121726 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:57.121451 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2" podUID="2bcaee75-ac15-4b13-8f60-401ee60464ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Nov 02 01:26:57.541501 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:57.541434 4427 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 02 01:26:57.541695 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:57.541674 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4" podUID="eb3258a9-adfb-46de-a0ab-7b4649948689" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 02 01:26:58.121428 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:58.121364 4427 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Nov 02 01:26:58.121608 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:58.121455 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2" podUID="2bcaee75-ac15-4b13-8f60-401ee60464ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Nov 02 01:26:58.542512 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:58.542407 4427 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 02 01:26:58.542512 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:58.542457 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4" podUID="eb3258a9-adfb-46de-a0ab-7b4649948689" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 02 01:26:59.122877 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:59.122838 4427 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Nov 02 01:26:59.123091 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:59.122901 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2" podUID="2bcaee75-ac15-4b13-8f60-401ee60464ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Nov 02 01:26:59.576912 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:59.576806 4427 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 02 01:26:59.577625 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:26:59.577469 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4" podUID="eb3258a9-adfb-46de-a0ab-7b4649948689" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 02 01:27:00.134985 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:00.134948 4427 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Nov 02 01:27:00.135300 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:00.135263 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2" podUID="2bcaee75-ac15-4b13-8f60-401ee60464ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Nov 02 01:27:00.541984 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:00.541546 4427 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 02 01:27:00.541984 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:00.541629 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4" podUID="eb3258a9-adfb-46de-a0ab-7b4649948689" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 02 01:27:01.122060 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:01.122020 4427 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Nov 02 01:27:01.122553 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:01.122091 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2" podUID="2bcaee75-ac15-4b13-8f60-401ee60464ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Nov 02 01:27:01.542835 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:01.542745 4427 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 02 01:27:01.542835 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:01.542801 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4" podUID="eb3258a9-adfb-46de-a0ab-7b4649948689" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 02 01:27:01.593314 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:27:01.593274 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:27:02.230187 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:02.230142 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3190c013-k92s4" event={"ID":"92b288ee-b9e7-494a-a48b-0452b09e9f03","Type":"ContainerStarted","Data":"8cf110a2d68b9d6d877d0a006337cde529667c6f5817791412fd15c54327fc33"} Nov 02 01:27:02.251985 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:02.251867 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3190c013-k92s4" podStartSLOduration=1.49152864 podStartE2EDuration="7.251846594s" totalImagesPullingTime="5.760317954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:26:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:27:02.24790479 +0000 UTC m=+726.868910004" watchObservedRunningTime="2026-11-02 01:27:02.251846594 +0000 UTC m=+726.872851817" Nov 02 01:27:02.378862 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:02.375876 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2" Nov 02 01:27:02.378862 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:02.376434 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2" Nov 02 01:27:02.409198 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:02.409165 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2" Nov 02 01:27:02.860639 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:02.860605 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4" Nov 02 01:27:02.861103 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:02.861074 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4" Nov 02 01:27:02.861975 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:02.861956 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4" Nov 02 01:27:03.237295 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:03.237260 4427 generic.go:356] "Generic (PLEG): container finished" podID="92b288ee-b9e7-494a-a48b-0452b09e9f03" containerID="8cf110a2d68b9d6d877d0a006337cde529667c6f5817791412fd15c54327fc33" exitCode=0 Nov 02 01:27:03.237781 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:03.237358 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3190c013-k92s4" event={"ID":"92b288ee-b9e7-494a-a48b-0452b09e9f03","Type":"ContainerDied","Data":"8cf110a2d68b9d6d877d0a006337cde529667c6f5817791412fd15c54327fc33"} Nov 02 01:27:04.410503 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:04.410474 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3190c013-k92s4" Nov 02 01:27:04.475720 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:04.475683 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92b288ee-b9e7-494a-a48b-0452b09e9f03-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92b288ee-b9e7-494a-a48b-0452b09e9f03-maas-api-xks-ca\") pod \"92b288ee-b9e7-494a-a48b-0452b09e9f03\" (UID: \"92b288ee-b9e7-494a-a48b-0452b09e9f03\") " Nov 02 01:27:04.475919 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:04.475845 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92b288ee-b9e7-494a-a48b-0452b09e9f03-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92b288ee-b9e7-494a-a48b-0452b09e9f03-maas-api-openshift-ca\") pod \"92b288ee-b9e7-494a-a48b-0452b09e9f03\" (UID: \"92b288ee-b9e7-494a-a48b-0452b09e9f03\") " Nov 02 01:27:04.475984 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:04.475947 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92b288ee-b9e7-494a-a48b-0452b09e9f03-kube-api-access-wpwqs\" (UniqueName: \"kubernetes.io/projected/92b288ee-b9e7-494a-a48b-0452b09e9f03-kube-api-access-wpwqs\") pod \"92b288ee-b9e7-494a-a48b-0452b09e9f03\" (UID: \"92b288ee-b9e7-494a-a48b-0452b09e9f03\") " Nov 02 01:27:04.476627 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:04.476595 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92b288ee-b9e7-494a-a48b-0452b09e9f03-maas-api-openshift-ca" pod "92b288ee-b9e7-494a-a48b-0452b09e9f03" (UID: "92b288ee-b9e7-494a-a48b-0452b09e9f03"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:27:04.478057 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:04.478028 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92b288ee-b9e7-494a-a48b-0452b09e9f03-maas-api-xks-ca" pod "92b288ee-b9e7-494a-a48b-0452b09e9f03" (UID: "92b288ee-b9e7-494a-a48b-0452b09e9f03"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:27:04.478733 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:04.478717 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92b288ee-b9e7-494a-a48b-0452b09e9f03-kube-api-access-wpwqs" pod "92b288ee-b9e7-494a-a48b-0452b09e9f03" (UID: "92b288ee-b9e7-494a-a48b-0452b09e9f03"). InnerVolumeSpecName "kube-api-access-wpwqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:27:04.577412 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:04.577304 4427 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92b288ee-b9e7-494a-a48b-0452b09e9f03-maas-api-xks-ca\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:27:04.577412 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:04.577348 4427 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92b288ee-b9e7-494a-a48b-0452b09e9f03-maas-api-openshift-ca\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:27:04.577412 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:04.577358 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpwqs\" (UniqueName: \"kubernetes.io/projected/92b288ee-b9e7-494a-a48b-0452b09e9f03-kube-api-access-wpwqs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:27:05.252155 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:05.252114 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3190c013-k92s4" event={"ID":"92b288ee-b9e7-494a-a48b-0452b09e9f03","Type":"ContainerDied","Data":"94ee40b61b171b54af455d18edf1c0fd1d97dabcbe10f7fae4b0e0a5468a34cd"} Nov 02 01:27:05.252326 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:05.252172 4427 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94ee40b61b171b54af455d18edf1c0fd1d97dabcbe10f7fae4b0e0a5468a34cd" Nov 02 01:27:05.252326 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:05.252124 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3190c013-k92s4" Nov 02 01:27:08.066231 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:08.066194 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96095420-4rp8x"] Nov 02 01:27:08.066712 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:08.066697 4427 cpu_manager.go:414] "containerMap: removing container" podUID="92b288ee-b9e7-494a-a48b-0452b09e9f03" containerName="revoke-keys" Nov 02 01:27:08.066752 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:08.066712 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92b288ee-b9e7-494a-a48b-0452b09e9f03" containerName="revoke-keys" Nov 02 01:27:08.066803 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:08.066794 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="92b288ee-b9e7-494a-a48b-0452b09e9f03" containerName="revoke-keys" Nov 02 01:27:08.070137 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:08.070121 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96095420-4rp8x" Nov 02 01:27:08.075893 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:08.075867 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 01:27:08.076682 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:08.076660 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\"" Nov 02 01:27:08.084026 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:08.083999 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96095420-4rp8x"] Nov 02 01:27:08.116432 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:08.116391 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67b125a9-f02b-416c-8108-e4a8a41f1001-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96095420-4rp8x\" (UID: \"67b125a9-f02b-416c-8108-e4a8a41f1001\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96095420-4rp8x" Nov 02 01:27:08.116606 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:08.116454 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpm8t\" (UniqueName: \"kubernetes.io/projected/67b125a9-f02b-416c-8108-e4a8a41f1001-kube-api-access-jpm8t\") pod \"maas-api-revoke-sub-96095420-4rp8x\" (UID: \"67b125a9-f02b-416c-8108-e4a8a41f1001\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96095420-4rp8x" Nov 02 01:27:08.116606 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:08.116570 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67b125a9-f02b-416c-8108-e4a8a41f1001-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96095420-4rp8x\" (UID: \"67b125a9-f02b-416c-8108-e4a8a41f1001\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96095420-4rp8x" Nov 02 01:27:08.217679 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:08.217627 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67b125a9-f02b-416c-8108-e4a8a41f1001-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96095420-4rp8x\" (UID: \"67b125a9-f02b-416c-8108-e4a8a41f1001\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96095420-4rp8x" Nov 02 01:27:08.217679 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:08.217685 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpm8t\" (UniqueName: \"kubernetes.io/projected/67b125a9-f02b-416c-8108-e4a8a41f1001-kube-api-access-jpm8t\") pod \"maas-api-revoke-sub-96095420-4rp8x\" (UID: \"67b125a9-f02b-416c-8108-e4a8a41f1001\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96095420-4rp8x" Nov 02 01:27:08.217955 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:08.217731 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67b125a9-f02b-416c-8108-e4a8a41f1001-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96095420-4rp8x\" (UID: \"67b125a9-f02b-416c-8108-e4a8a41f1001\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96095420-4rp8x" Nov 02 01:27:08.218365 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:08.218346 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67b125a9-f02b-416c-8108-e4a8a41f1001-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96095420-4rp8x\" (UID: \"67b125a9-f02b-416c-8108-e4a8a41f1001\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96095420-4rp8x" Nov 02 01:27:08.220099 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:08.220077 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67b125a9-f02b-416c-8108-e4a8a41f1001-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96095420-4rp8x\" (UID: \"67b125a9-f02b-416c-8108-e4a8a41f1001\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96095420-4rp8x" Nov 02 01:27:08.233007 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:08.232983 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpm8t\" (UniqueName: \"kubernetes.io/projected/67b125a9-f02b-416c-8108-e4a8a41f1001-kube-api-access-jpm8t\") pod \"maas-api-revoke-sub-96095420-4rp8x\" (UID: \"67b125a9-f02b-416c-8108-e4a8a41f1001\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96095420-4rp8x" Nov 02 01:27:08.386359 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:08.386326 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96095420-4rp8x" Nov 02 01:27:08.675962 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:08.675930 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96095420-4rp8x"] Nov 02 01:27:09.275046 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:09.274956 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96095420-4rp8x" event={"ID":"67b125a9-f02b-416c-8108-e4a8a41f1001","Type":"ContainerStarted","Data":"0478aca3b5c933fc6d216abc2c10806a4f308c3183c001fc306f56c6b257f3e5"} Nov 02 01:27:09.275046 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:09.275015 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96095420-4rp8x" event={"ID":"67b125a9-f02b-416c-8108-e4a8a41f1001","Type":"ContainerStarted","Data":"1499d0c3481919d2137f66f6a86826062ec5893eea2bf4382df918d96c1e0aec"} Nov 02 01:27:09.294984 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:09.294936 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96095420-4rp8x" podStartSLOduration=1.294921419 podStartE2EDuration="1.294921419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:27:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:27:09.293525587 +0000 UTC m=+733.914530798" watchObservedRunningTime="2026-11-02 01:27:09.294921419 +0000 UTC m=+733.915926630" Nov 02 01:27:10.282504 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:10.282464 4427 generic.go:356] "Generic (PLEG): container finished" podID="67b125a9-f02b-416c-8108-e4a8a41f1001" containerID="0478aca3b5c933fc6d216abc2c10806a4f308c3183c001fc306f56c6b257f3e5" exitCode=0 Nov 02 01:27:10.282932 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:10.282559 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96095420-4rp8x" event={"ID":"67b125a9-f02b-416c-8108-e4a8a41f1001","Type":"ContainerDied","Data":"0478aca3b5c933fc6d216abc2c10806a4f308c3183c001fc306f56c6b257f3e5"} Nov 02 01:27:12.169483 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:12.155598 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96095420-4rp8x" Nov 02 01:27:12.280398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:12.273884 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67b125a9-f02b-416c-8108-e4a8a41f1001-kube-api-access-jpm8t\" (UniqueName: \"kubernetes.io/projected/67b125a9-f02b-416c-8108-e4a8a41f1001-kube-api-access-jpm8t\") pod \"67b125a9-f02b-416c-8108-e4a8a41f1001\" (UID: \"67b125a9-f02b-416c-8108-e4a8a41f1001\") " Nov 02 01:27:12.280398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:12.274017 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67b125a9-f02b-416c-8108-e4a8a41f1001-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67b125a9-f02b-416c-8108-e4a8a41f1001-maas-api-openshift-ca\") pod \"67b125a9-f02b-416c-8108-e4a8a41f1001\" (UID: \"67b125a9-f02b-416c-8108-e4a8a41f1001\") " Nov 02 01:27:12.280398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:12.274114 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67b125a9-f02b-416c-8108-e4a8a41f1001-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67b125a9-f02b-416c-8108-e4a8a41f1001-maas-api-xks-ca\") pod \"67b125a9-f02b-416c-8108-e4a8a41f1001\" (UID: \"67b125a9-f02b-416c-8108-e4a8a41f1001\") " Nov 02 01:27:12.280398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:12.275787 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67b125a9-f02b-416c-8108-e4a8a41f1001-maas-api-openshift-ca" pod "67b125a9-f02b-416c-8108-e4a8a41f1001" (UID: "67b125a9-f02b-416c-8108-e4a8a41f1001"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:27:12.296948 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:12.294533 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67b125a9-f02b-416c-8108-e4a8a41f1001-maas-api-xks-ca" pod "67b125a9-f02b-416c-8108-e4a8a41f1001" (UID: "67b125a9-f02b-416c-8108-e4a8a41f1001"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:27:12.297645 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:12.297607 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67b125a9-f02b-416c-8108-e4a8a41f1001-kube-api-access-jpm8t" pod "67b125a9-f02b-416c-8108-e4a8a41f1001" (UID: "67b125a9-f02b-416c-8108-e4a8a41f1001"). InnerVolumeSpecName "kube-api-access-jpm8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:27:12.332967 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:12.332925 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96095420-4rp8x" event={"ID":"67b125a9-f02b-416c-8108-e4a8a41f1001","Type":"ContainerDied","Data":"1499d0c3481919d2137f66f6a86826062ec5893eea2bf4382df918d96c1e0aec"} Nov 02 01:27:12.333274 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:12.332958 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96095420-4rp8x" Nov 02 01:27:12.333940 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:12.333257 4427 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1499d0c3481919d2137f66f6a86826062ec5893eea2bf4382df918d96c1e0aec" Nov 02 01:27:12.376106 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:12.376071 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpm8t\" (UniqueName: \"kubernetes.io/projected/67b125a9-f02b-416c-8108-e4a8a41f1001-kube-api-access-jpm8t\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:27:12.376355 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:12.376334 4427 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67b125a9-f02b-416c-8108-e4a8a41f1001-maas-api-openshift-ca\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:27:12.376525 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:12.376508 4427 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67b125a9-f02b-416c-8108-e4a8a41f1001-maas-api-xks-ca\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:27:31.583141 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:27:31.583103 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:27:32.510646 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:32.510586 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b"] Nov 02 01:27:32.511803 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:32.511769 4427 cpu_manager.go:414] "containerMap: removing container" podUID="67b125a9-f02b-416c-8108-e4a8a41f1001" containerName="revoke-keys" Nov 02 01:27:32.511803 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:32.511798 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67b125a9-f02b-416c-8108-e4a8a41f1001" containerName="revoke-keys" Nov 02 01:27:32.512009 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:32.511926 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="67b125a9-f02b-416c-8108-e4a8a41f1001" containerName="revoke-keys" Nov 02 01:27:32.521272 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:32.521247 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b"] Nov 02 01:27:32.521451 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:32.521355 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" Nov 02 01:27:32.527694 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:32.527655 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-47dd19\"/\"distinct-w2-47dd19-kserve-self-signed-certs\"" Nov 02 01:27:32.548115 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:32.548084 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea61971c-5a72-4609-b5a0-58a74b324bb7-dshm\") pod \"distinct-w2-47dd19-kserve-95d95b88d-5m78b\" (UID: \"ea61971c-5a72-4609-b5a0-58a74b324bb7\") " pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" Nov 02 01:27:32.548278 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:32.548172 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea61971c-5a72-4609-b5a0-58a74b324bb7-home\") pod \"distinct-w2-47dd19-kserve-95d95b88d-5m78b\" (UID: \"ea61971c-5a72-4609-b5a0-58a74b324bb7\") " pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" Nov 02 01:27:32.548328 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:32.548279 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea61971c-5a72-4609-b5a0-58a74b324bb7-tmp-dir\") pod \"distinct-w2-47dd19-kserve-95d95b88d-5m78b\" (UID: \"ea61971c-5a72-4609-b5a0-58a74b324bb7\") " pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" Nov 02 01:27:32.548419 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:32.548364 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea61971c-5a72-4609-b5a0-58a74b324bb7-model-cache\") pod \"distinct-w2-47dd19-kserve-95d95b88d-5m78b\" (UID: \"ea61971c-5a72-4609-b5a0-58a74b324bb7\") " pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" Nov 02 01:27:32.548594 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:32.548574 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw7gl\" (UniqueName: \"kubernetes.io/projected/ea61971c-5a72-4609-b5a0-58a74b324bb7-kube-api-access-hw7gl\") pod \"distinct-w2-47dd19-kserve-95d95b88d-5m78b\" (UID: \"ea61971c-5a72-4609-b5a0-58a74b324bb7\") " pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" Nov 02 01:27:32.548637 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:32.548626 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea61971c-5a72-4609-b5a0-58a74b324bb7-tls-certs\") pod \"distinct-w2-47dd19-kserve-95d95b88d-5m78b\" (UID: \"ea61971c-5a72-4609-b5a0-58a74b324bb7\") " pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" Nov 02 01:27:32.650503 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:32.650466 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea61971c-5a72-4609-b5a0-58a74b324bb7-dshm\") pod \"distinct-w2-47dd19-kserve-95d95b88d-5m78b\" (UID: \"ea61971c-5a72-4609-b5a0-58a74b324bb7\") " pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" Nov 02 01:27:32.651150 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:32.651126 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea61971c-5a72-4609-b5a0-58a74b324bb7-home\") pod \"distinct-w2-47dd19-kserve-95d95b88d-5m78b\" (UID: \"ea61971c-5a72-4609-b5a0-58a74b324bb7\") " pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" Nov 02 01:27:32.652523 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:32.651664 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea61971c-5a72-4609-b5a0-58a74b324bb7-home\") pod \"distinct-w2-47dd19-kserve-95d95b88d-5m78b\" (UID: \"ea61971c-5a72-4609-b5a0-58a74b324bb7\") " pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" Nov 02 01:27:32.652523 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:32.651752 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea61971c-5a72-4609-b5a0-58a74b324bb7-tmp-dir\") pod \"distinct-w2-47dd19-kserve-95d95b88d-5m78b\" (UID: \"ea61971c-5a72-4609-b5a0-58a74b324bb7\") " pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" Nov 02 01:27:32.652523 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:32.651847 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea61971c-5a72-4609-b5a0-58a74b324bb7-model-cache\") pod \"distinct-w2-47dd19-kserve-95d95b88d-5m78b\" (UID: \"ea61971c-5a72-4609-b5a0-58a74b324bb7\") " pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" Nov 02 01:27:32.652523 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:32.651885 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw7gl\" (UniqueName: \"kubernetes.io/projected/ea61971c-5a72-4609-b5a0-58a74b324bb7-kube-api-access-hw7gl\") pod \"distinct-w2-47dd19-kserve-95d95b88d-5m78b\" (UID: \"ea61971c-5a72-4609-b5a0-58a74b324bb7\") " pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" Nov 02 01:27:32.652523 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:32.651928 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea61971c-5a72-4609-b5a0-58a74b324bb7-tls-certs\") pod \"distinct-w2-47dd19-kserve-95d95b88d-5m78b\" (UID: \"ea61971c-5a72-4609-b5a0-58a74b324bb7\") " pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" Nov 02 01:27:32.652523 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:32.652017 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea61971c-5a72-4609-b5a0-58a74b324bb7-tmp-dir\") pod \"distinct-w2-47dd19-kserve-95d95b88d-5m78b\" (UID: \"ea61971c-5a72-4609-b5a0-58a74b324bb7\") " pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" Nov 02 01:27:32.652523 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:32.652336 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea61971c-5a72-4609-b5a0-58a74b324bb7-model-cache\") pod \"distinct-w2-47dd19-kserve-95d95b88d-5m78b\" (UID: \"ea61971c-5a72-4609-b5a0-58a74b324bb7\") " pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" Nov 02 01:27:32.656917 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:32.656893 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea61971c-5a72-4609-b5a0-58a74b324bb7-tls-certs\") pod \"distinct-w2-47dd19-kserve-95d95b88d-5m78b\" (UID: \"ea61971c-5a72-4609-b5a0-58a74b324bb7\") " pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" Nov 02 01:27:32.657111 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:32.657091 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea61971c-5a72-4609-b5a0-58a74b324bb7-dshm\") pod \"distinct-w2-47dd19-kserve-95d95b88d-5m78b\" (UID: \"ea61971c-5a72-4609-b5a0-58a74b324bb7\") " pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" Nov 02 01:27:32.671728 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:32.671698 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw7gl\" (UniqueName: \"kubernetes.io/projected/ea61971c-5a72-4609-b5a0-58a74b324bb7-kube-api-access-hw7gl\") pod \"distinct-w2-47dd19-kserve-95d95b88d-5m78b\" (UID: \"ea61971c-5a72-4609-b5a0-58a74b324bb7\") " pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" Nov 02 01:27:32.840084 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:32.839991 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" Nov 02 01:27:33.124956 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:33.124922 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b"] Nov 02 01:27:33.127220 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:27:33.127191 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea61971c_5a72_4609_b5a0_58a74b324bb7.slice/crio-6ec71cadf23ae919ed1c23ca0d12d32a998ffefd8b6f4742db6b83fdab86e10d WatchSource:0}: Error finding container 6ec71cadf23ae919ed1c23ca0d12d32a998ffefd8b6f4742db6b83fdab86e10d: Status 404 returned error can't find the container with id 6ec71cadf23ae919ed1c23ca0d12d32a998ffefd8b6f4742db6b83fdab86e10d Nov 02 01:27:33.489029 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:33.488944 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" event={"ID":"ea61971c-5a72-4609-b5a0-58a74b324bb7","Type":"ContainerStarted","Data":"e8a579faafe103c3a9a2216d88d2dcba9caa4e82a8e7784965ef4d349389d44c"} Nov 02 01:27:33.489029 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:33.488991 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" event={"ID":"ea61971c-5a72-4609-b5a0-58a74b324bb7","Type":"ContainerStarted","Data":"6ec71cadf23ae919ed1c23ca0d12d32a998ffefd8b6f4742db6b83fdab86e10d"} Nov 02 01:27:33.510244 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:33.510188 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" podStartSLOduration=1.510172124 podStartE2EDuration="1.510172124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:27:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:27:33.507645429 +0000 UTC m=+758.128650641" watchObservedRunningTime="2026-11-02 01:27:33.510172124 +0000 UTC m=+758.131177336" Nov 02 01:27:33.840936 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:33.840844 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" Nov 02 01:27:42.873012 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:42.868485 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" Nov 02 01:27:43.148982 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:43.148892 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" Nov 02 01:27:43.611144 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:43.611118 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" Nov 02 01:27:46.455892 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:46.455855 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr"] Nov 02 01:27:46.460323 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:46.460290 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" Nov 02 01:27:46.471064 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:46.470687 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ca1c53\"/\"kube-root-ca.crt\"" Nov 02 01:27:46.471064 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:46.470773 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ca1c53\"/\"openshift-service-ca.crt\"" Nov 02 01:27:46.471287 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:46.471259 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr"] Nov 02 01:27:46.472043 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:46.471998 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ca1c53\"/\"default-dockercfg-s6h7m\"" Nov 02 01:27:46.472186 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:46.472089 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ca1c53\"/\"distinct-w0-ca1c53-kserve-self-signed-certs\"" Nov 02 01:27:46.545462 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:46.545427 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c056798-2cf8-47b1-9829-67cebda7a248-dshm\") pod \"distinct-w0-ca1c53-kserve-579fcb65df-j95qr\" (UID: \"9c056798-2cf8-47b1-9829-67cebda7a248\") " pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" Nov 02 01:27:46.545674 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:46.545485 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c056798-2cf8-47b1-9829-67cebda7a248-model-cache\") pod \"distinct-w0-ca1c53-kserve-579fcb65df-j95qr\" (UID: \"9c056798-2cf8-47b1-9829-67cebda7a248\") " pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" Nov 02 01:27:46.545674 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:46.545640 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c056798-2cf8-47b1-9829-67cebda7a248-tmp-dir\") pod \"distinct-w0-ca1c53-kserve-579fcb65df-j95qr\" (UID: \"9c056798-2cf8-47b1-9829-67cebda7a248\") " pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" Nov 02 01:27:46.545796 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:46.545683 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c056798-2cf8-47b1-9829-67cebda7a248-tls-certs\") pod \"distinct-w0-ca1c53-kserve-579fcb65df-j95qr\" (UID: \"9c056798-2cf8-47b1-9829-67cebda7a248\") " pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" Nov 02 01:27:46.545845 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:46.545791 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57sbh\" (UniqueName: \"kubernetes.io/projected/9c056798-2cf8-47b1-9829-67cebda7a248-kube-api-access-57sbh\") pod \"distinct-w0-ca1c53-kserve-579fcb65df-j95qr\" (UID: \"9c056798-2cf8-47b1-9829-67cebda7a248\") " pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" Nov 02 01:27:46.545893 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:46.545845 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c056798-2cf8-47b1-9829-67cebda7a248-home\") pod \"distinct-w0-ca1c53-kserve-579fcb65df-j95qr\" (UID: \"9c056798-2cf8-47b1-9829-67cebda7a248\") " pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" Nov 02 01:27:46.647149 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:46.647095 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57sbh\" (UniqueName: \"kubernetes.io/projected/9c056798-2cf8-47b1-9829-67cebda7a248-kube-api-access-57sbh\") pod \"distinct-w0-ca1c53-kserve-579fcb65df-j95qr\" (UID: \"9c056798-2cf8-47b1-9829-67cebda7a248\") " pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" Nov 02 01:27:46.647435 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:46.647180 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c056798-2cf8-47b1-9829-67cebda7a248-home\") pod \"distinct-w0-ca1c53-kserve-579fcb65df-j95qr\" (UID: \"9c056798-2cf8-47b1-9829-67cebda7a248\") " pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" Nov 02 01:27:46.647435 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:46.647291 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c056798-2cf8-47b1-9829-67cebda7a248-dshm\") pod \"distinct-w0-ca1c53-kserve-579fcb65df-j95qr\" (UID: \"9c056798-2cf8-47b1-9829-67cebda7a248\") " pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" Nov 02 01:27:46.647435 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:46.647407 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c056798-2cf8-47b1-9829-67cebda7a248-model-cache\") pod \"distinct-w0-ca1c53-kserve-579fcb65df-j95qr\" (UID: \"9c056798-2cf8-47b1-9829-67cebda7a248\") " pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" Nov 02 01:27:46.647631 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:46.647511 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c056798-2cf8-47b1-9829-67cebda7a248-tmp-dir\") pod \"distinct-w0-ca1c53-kserve-579fcb65df-j95qr\" (UID: \"9c056798-2cf8-47b1-9829-67cebda7a248\") " pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" Nov 02 01:27:46.647631 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:46.647538 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c056798-2cf8-47b1-9829-67cebda7a248-tls-certs\") pod \"distinct-w0-ca1c53-kserve-579fcb65df-j95qr\" (UID: \"9c056798-2cf8-47b1-9829-67cebda7a248\") " pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" Nov 02 01:27:46.656571 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:46.656531 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c056798-2cf8-47b1-9829-67cebda7a248-tls-certs\") pod \"distinct-w0-ca1c53-kserve-579fcb65df-j95qr\" (UID: \"9c056798-2cf8-47b1-9829-67cebda7a248\") " pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" Nov 02 01:27:46.657054 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:46.657034 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c056798-2cf8-47b1-9829-67cebda7a248-tmp-dir\") pod \"distinct-w0-ca1c53-kserve-579fcb65df-j95qr\" (UID: \"9c056798-2cf8-47b1-9829-67cebda7a248\") " pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" Nov 02 01:27:46.658628 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:46.658605 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c056798-2cf8-47b1-9829-67cebda7a248-model-cache\") pod \"distinct-w0-ca1c53-kserve-579fcb65df-j95qr\" (UID: \"9c056798-2cf8-47b1-9829-67cebda7a248\") " pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" Nov 02 01:27:46.659925 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:46.659900 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c056798-2cf8-47b1-9829-67cebda7a248-dshm\") pod \"distinct-w0-ca1c53-kserve-579fcb65df-j95qr\" (UID: \"9c056798-2cf8-47b1-9829-67cebda7a248\") " pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" Nov 02 01:27:46.661650 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:46.661574 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c056798-2cf8-47b1-9829-67cebda7a248-home\") pod \"distinct-w0-ca1c53-kserve-579fcb65df-j95qr\" (UID: \"9c056798-2cf8-47b1-9829-67cebda7a248\") " pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" Nov 02 01:27:46.667898 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:46.667069 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57sbh\" (UniqueName: \"kubernetes.io/projected/9c056798-2cf8-47b1-9829-67cebda7a248-kube-api-access-57sbh\") pod \"distinct-w0-ca1c53-kserve-579fcb65df-j95qr\" (UID: \"9c056798-2cf8-47b1-9829-67cebda7a248\") " pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" Nov 02 01:27:46.783221 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:46.783130 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" Nov 02 01:27:47.071105 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:47.070994 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr"] Nov 02 01:27:47.079430 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:27:47.079362 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c056798_2cf8_47b1_9829_67cebda7a248.slice/crio-0f24700eda7f7fc576960de14a0fd9c7277d5b5eea444e28d5af54097e1f5775 WatchSource:0}: Error finding container 0f24700eda7f7fc576960de14a0fd9c7277d5b5eea444e28d5af54097e1f5775: Status 404 returned error can't find the container with id 0f24700eda7f7fc576960de14a0fd9c7277d5b5eea444e28d5af54097e1f5775 Nov 02 01:27:47.643039 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:47.642997 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" event={"ID":"9c056798-2cf8-47b1-9829-67cebda7a248","Type":"ContainerStarted","Data":"f25cae758ce8cb349de3a1db3abe515c731d1858d620b5f27f21d0be55ee4a41"} Nov 02 01:27:47.643039 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:47.643057 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" event={"ID":"9c056798-2cf8-47b1-9829-67cebda7a248","Type":"ContainerStarted","Data":"0f24700eda7f7fc576960de14a0fd9c7277d5b5eea444e28d5af54097e1f5775"} Nov 02 01:27:47.662896 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:47.662854 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" podStartSLOduration=1.662839113 podStartE2EDuration="1.662839113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:27:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:27:47.662114868 +0000 UTC m=+772.283120080" watchObservedRunningTime="2026-11-02 01:27:47.662839113 +0000 UTC m=+772.283844324" Nov 02 01:27:47.784081 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:47.784039 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" Nov 02 01:27:54.321398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:54.318878 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv"] Nov 02 01:27:54.332252 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:54.325510 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv" Nov 02 01:27:54.332252 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:54.327928 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2"] Nov 02 01:27:54.350633 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:54.350606 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2" Nov 02 01:27:54.411399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:54.353297 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv"] Nov 02 01:27:54.411399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:54.397297 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2"] Nov 02 01:27:54.498812 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:54.498766 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/819ae12d-e493-42d0-a549-811fb5fe7499-workload-certs\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv\" (UID: \"819ae12d-e493-42d0-a549-811fb5fe7499\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv" Nov 02 01:27:54.500199 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:54.500168 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n94kc\" (UniqueName: \"kubernetes.io/projected/165f361d-6b06-4c6c-8598-27c4b522b2e2-kube-api-access-n94kc\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2\" (UID: \"165f361d-6b06-4c6c-8598-27c4b522b2e2\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2" Nov 02 01:27:54.500455 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:54.500436 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/819ae12d-e493-42d0-a549-811fb5fe7499-credential-socket\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv\" (UID: \"819ae12d-e493-42d0-a549-811fb5fe7499\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv" Nov 02 01:27:54.500640 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:54.500622 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/819ae12d-e493-42d0-a549-811fb5fe7499-istio-envoy\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv\" (UID: \"819ae12d-e493-42d0-a549-811fb5fe7499\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv" Nov 02 01:27:54.500818 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:54.500802 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/165f361d-6b06-4c6c-8598-27c4b522b2e2-istio-podinfo\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2\" (UID: \"165f361d-6b06-4c6c-8598-27c4b522b2e2\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2" Nov 02 01:27:54.500982 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:54.500966 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/165f361d-6b06-4c6c-8598-27c4b522b2e2-istio-data\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2\" (UID: \"165f361d-6b06-4c6c-8598-27c4b522b2e2\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2" Nov 02 01:27:54.501174 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:54.501155 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/165f361d-6b06-4c6c-8598-27c4b522b2e2-credential-socket\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2\" (UID: \"165f361d-6b06-4c6c-8598-27c4b522b2e2\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2" Nov 02 01:27:54.501341 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:54.501324 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/819ae12d-e493-42d0-a549-811fb5fe7499-istio-podinfo\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv\" (UID: \"819ae12d-e493-42d0-a549-811fb5fe7499\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv" Nov 02 01:27:54.501548 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:54.501528 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/819ae12d-e493-42d0-a549-811fb5fe7499-workload-socket\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv\" (UID: \"819ae12d-e493-42d0-a549-811fb5fe7499\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv" Nov 02 01:27:54.501697 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:54.501681 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/819ae12d-e493-42d0-a549-811fb5fe7499-istio-data\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv\" (UID: \"819ae12d-e493-42d0-a549-811fb5fe7499\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv" Nov 02 01:27:54.501874 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:54.501857 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/165f361d-6b06-4c6c-8598-27c4b522b2e2-workload-certs\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2\" (UID: \"165f361d-6b06-4c6c-8598-27c4b522b2e2\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2" Nov 02 01:27:54.502024 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:54.502008 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/819ae12d-e493-42d0-a549-811fb5fe7499-istio-token\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv\" (UID: \"819ae12d-e493-42d0-a549-811fb5fe7499\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv" Nov 02 01:27:54.502180 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:54.502162 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/819ae12d-e493-42d0-a549-811fb5fe7499-istiod-ca-cert\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv\" (UID: \"819ae12d-e493-42d0-a549-811fb5fe7499\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv" Nov 02 01:27:54.502353 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:54.502336 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/165f361d-6b06-4c6c-8598-27c4b522b2e2-istio-envoy\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2\" (UID: \"165f361d-6b06-4c6c-8598-27c4b522b2e2\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2" Nov 02 01:27:54.502614 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:54.502594 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/165f361d-6b06-4c6c-8598-27c4b522b2e2-istio-token\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2\" (UID: \"165f361d-6b06-4c6c-8598-27c4b522b2e2\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2" Nov 02 01:27:54.502781 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:54.502763 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/165f361d-6b06-4c6c-8598-27c4b522b2e2-istiod-ca-cert\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2\" (UID: \"165f361d-6b06-4c6c-8598-27c4b522b2e2\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2" Nov 02 01:27:54.502951 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:54.502934 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/165f361d-6b06-4c6c-8598-27c4b522b2e2-workload-socket\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2\" (UID: \"165f361d-6b06-4c6c-8598-27c4b522b2e2\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2" Nov 02 01:27:54.503153 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:54.503136 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jh6k\" (UniqueName: \"kubernetes.io/projected/819ae12d-e493-42d0-a549-811fb5fe7499-kube-api-access-6jh6k\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv\" (UID: \"819ae12d-e493-42d0-a549-811fb5fe7499\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv" Nov 02 01:27:54.606070 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:54.605985 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jh6k\" (UniqueName: \"kubernetes.io/projected/819ae12d-e493-42d0-a549-811fb5fe7499-kube-api-access-6jh6k\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv\" (UID: \"819ae12d-e493-42d0-a549-811fb5fe7499\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv" Nov 02 01:27:54.606796 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:54.606776 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/819ae12d-e493-42d0-a549-811fb5fe7499-workload-certs\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv\" (UID: \"819ae12d-e493-42d0-a549-811fb5fe7499\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv" Nov 02 01:27:54.607398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:54.607284 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/819ae12d-e493-42d0-a549-811fb5fe7499-workload-certs\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv\" (UID: \"819ae12d-e493-42d0-a549-811fb5fe7499\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv" Nov 02 01:27:54.607522 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:54.607355 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n94kc\" (UniqueName: \"kubernetes.io/projected/165f361d-6b06-4c6c-8598-27c4b522b2e2-kube-api-access-n94kc\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2\" (UID: \"165f361d-6b06-4c6c-8598-27c4b522b2e2\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2" Nov 02 01:27:54.607866 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:54.607848 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/819ae12d-e493-42d0-a549-811fb5fe7499-credential-socket\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv\" (UID: \"819ae12d-e493-42d0-a549-811fb5fe7499\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv" Nov 02 01:27:54.608292 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:54.608194 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/819ae12d-e493-42d0-a549-811fb5fe7499-credential-socket\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv\" (UID: \"819ae12d-e493-42d0-a549-811fb5fe7499\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv" Nov 02 01:27:54.608445 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:54.608429 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/819ae12d-e493-42d0-a549-811fb5fe7499-istio-envoy\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv\" (UID: \"819ae12d-e493-42d0-a549-811fb5fe7499\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv" Nov 02 01:27:54.608878 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:54.608860 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/165f361d-6b06-4c6c-8598-27c4b522b2e2-istio-podinfo\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2\" (UID: \"165f361d-6b06-4c6c-8598-27c4b522b2e2\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2" Nov 02 01:27:54.609402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:54.609365 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/165f361d-6b06-4c6c-8598-27c4b522b2e2-istio-data\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2\" (UID: \"165f361d-6b06-4c6c-8598-27c4b522b2e2\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2" Nov 02 01:27:54.609900 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:54.609885 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/165f361d-6b06-4c6c-8598-27c4b522b2e2-credential-socket\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2\" (UID: \"165f361d-6b06-4c6c-8598-27c4b522b2e2\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2" Nov 02 01:27:54.610024 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:54.610010 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/819ae12d-e493-42d0-a549-811fb5fe7499-istio-podinfo\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv\" (UID: \"819ae12d-e493-42d0-a549-811fb5fe7499\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv" Nov 02 01:27:54.610148 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:54.610136 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/819ae12d-e493-42d0-a549-811fb5fe7499-workload-socket\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv\" (UID: \"819ae12d-e493-42d0-a549-811fb5fe7499\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv" Nov 02 01:27:54.610236 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:54.610224 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/819ae12d-e493-42d0-a549-811fb5fe7499-istio-data\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv\" (UID: \"819ae12d-e493-42d0-a549-811fb5fe7499\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv" Nov 02 01:27:54.610336 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:54.610323 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/165f361d-6b06-4c6c-8598-27c4b522b2e2-workload-certs\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2\" (UID: \"165f361d-6b06-4c6c-8598-27c4b522b2e2\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2" Nov 02 01:27:54.611656 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:54.611640 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/819ae12d-e493-42d0-a549-811fb5fe7499-istio-token\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv\" (UID: \"819ae12d-e493-42d0-a549-811fb5fe7499\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv" Nov 02 01:27:54.612010 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:54.611994 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/819ae12d-e493-42d0-a549-811fb5fe7499-istiod-ca-cert\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv\" (UID: \"819ae12d-e493-42d0-a549-811fb5fe7499\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv" Nov 02 01:27:54.612798 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:54.612780 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/165f361d-6b06-4c6c-8598-27c4b522b2e2-istio-envoy\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2\" (UID: \"165f361d-6b06-4c6c-8598-27c4b522b2e2\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2" Nov 02 01:27:54.612965 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:54.612951 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/165f361d-6b06-4c6c-8598-27c4b522b2e2-istio-token\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2\" (UID: \"165f361d-6b06-4c6c-8598-27c4b522b2e2\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2" Nov 02 01:27:54.613070 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:54.613058 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/165f361d-6b06-4c6c-8598-27c4b522b2e2-istiod-ca-cert\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2\" (UID: \"165f361d-6b06-4c6c-8598-27c4b522b2e2\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2" Nov 02 01:27:54.613263 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:54.613246 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/165f361d-6b06-4c6c-8598-27c4b522b2e2-workload-socket\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2\" (UID: \"165f361d-6b06-4c6c-8598-27c4b522b2e2\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2" Nov 02 01:27:54.613599 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:54.611122 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/819ae12d-e493-42d0-a549-811fb5fe7499-workload-socket\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv\" (UID: \"819ae12d-e493-42d0-a549-811fb5fe7499\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv" Nov 02 01:27:54.627429 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:54.612717 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/819ae12d-e493-42d0-a549-811fb5fe7499-istiod-ca-cert\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv\" (UID: \"819ae12d-e493-42d0-a549-811fb5fe7499\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv" Nov 02 01:27:54.627593 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:54.611566 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/165f361d-6b06-4c6c-8598-27c4b522b2e2-workload-certs\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2\" (UID: \"165f361d-6b06-4c6c-8598-27c4b522b2e2\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2" Nov 02 01:27:54.627677 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:54.611329 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/819ae12d-e493-42d0-a549-811fb5fe7499-istio-data\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv\" (UID: \"819ae12d-e493-42d0-a549-811fb5fe7499\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv" Nov 02 01:27:54.627755 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:54.609804 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/165f361d-6b06-4c6c-8598-27c4b522b2e2-istio-data\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2\" (UID: \"165f361d-6b06-4c6c-8598-27c4b522b2e2\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2" Nov 02 01:27:54.627833 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:54.619755 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/165f361d-6b06-4c6c-8598-27c4b522b2e2-workload-socket\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2\" (UID: \"165f361d-6b06-4c6c-8598-27c4b522b2e2\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2" Nov 02 01:27:54.627909 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:54.610905 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/165f361d-6b06-4c6c-8598-27c4b522b2e2-credential-socket\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2\" (UID: \"165f361d-6b06-4c6c-8598-27c4b522b2e2\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2" Nov 02 01:27:54.627981 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:54.624657 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/165f361d-6b06-4c6c-8598-27c4b522b2e2-istio-envoy\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2\" (UID: \"165f361d-6b06-4c6c-8598-27c4b522b2e2\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2" Nov 02 01:27:54.628054 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:54.626746 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/165f361d-6b06-4c6c-8598-27c4b522b2e2-istiod-ca-cert\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2\" (UID: \"165f361d-6b06-4c6c-8598-27c4b522b2e2\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2" Nov 02 01:27:54.635684 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:54.635662 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n94kc\" (UniqueName: \"kubernetes.io/projected/165f361d-6b06-4c6c-8598-27c4b522b2e2-kube-api-access-n94kc\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2\" (UID: \"165f361d-6b06-4c6c-8598-27c4b522b2e2\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2" Nov 02 01:27:54.649757 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:54.646950 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/165f361d-6b06-4c6c-8598-27c4b522b2e2-istio-podinfo\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2\" (UID: \"165f361d-6b06-4c6c-8598-27c4b522b2e2\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2" Nov 02 01:27:54.649757 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:54.648032 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/165f361d-6b06-4c6c-8598-27c4b522b2e2-istio-token\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2\" (UID: \"165f361d-6b06-4c6c-8598-27c4b522b2e2\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2" Nov 02 01:27:54.649757 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:54.648806 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jh6k\" (UniqueName: \"kubernetes.io/projected/819ae12d-e493-42d0-a549-811fb5fe7499-kube-api-access-6jh6k\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv\" (UID: \"819ae12d-e493-42d0-a549-811fb5fe7499\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv" Nov 02 01:27:54.665406 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:54.662717 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/819ae12d-e493-42d0-a549-811fb5fe7499-istio-envoy\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv\" (UID: \"819ae12d-e493-42d0-a549-811fb5fe7499\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv" Nov 02 01:27:54.677400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:54.674248 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/819ae12d-e493-42d0-a549-811fb5fe7499-istio-podinfo\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv\" (UID: \"819ae12d-e493-42d0-a549-811fb5fe7499\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv" Nov 02 01:27:54.694457 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:54.678020 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/819ae12d-e493-42d0-a549-811fb5fe7499-istio-token\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv\" (UID: \"819ae12d-e493-42d0-a549-811fb5fe7499\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv" Nov 02 01:27:54.722496 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:54.722466 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2" Nov 02 01:27:54.723975 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:54.723794 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv" Nov 02 01:27:55.079406 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:55.079148 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7"] Nov 02 01:27:55.152846 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:55.148686 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7"] Nov 02 01:27:55.153015 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:55.148828 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" Nov 02 01:27:55.160754 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:55.160717 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b6ec07\"/\"default-dockercfg-s95bt\"" Nov 02 01:27:55.268768 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:55.190347 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/353a8a41-21ee-4470-861c-b53833beb106-model-cache\") pod \"distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7\" (UID: \"353a8a41-21ee-4470-861c-b53833beb106\") " pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" Nov 02 01:27:55.268768 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:55.190449 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/353a8a41-21ee-4470-861c-b53833beb106-dshm\") pod \"distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7\" (UID: \"353a8a41-21ee-4470-861c-b53833beb106\") " pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" Nov 02 01:27:55.268768 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:55.190524 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/353a8a41-21ee-4470-861c-b53833beb106-tmp-dir\") pod \"distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7\" (UID: \"353a8a41-21ee-4470-861c-b53833beb106\") " pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" Nov 02 01:27:55.268768 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:55.190641 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm2p9\" (UniqueName: \"kubernetes.io/projected/353a8a41-21ee-4470-861c-b53833beb106-kube-api-access-zm2p9\") pod \"distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7\" (UID: \"353a8a41-21ee-4470-861c-b53833beb106\") " pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" Nov 02 01:27:55.268768 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:55.190758 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/353a8a41-21ee-4470-861c-b53833beb106-home\") pod \"distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7\" (UID: \"353a8a41-21ee-4470-861c-b53833beb106\") " pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" Nov 02 01:27:55.268768 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:55.190791 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/353a8a41-21ee-4470-861c-b53833beb106-tls-certs\") pod \"distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7\" (UID: \"353a8a41-21ee-4470-861c-b53833beb106\") " pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" Nov 02 01:27:55.268768 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:55.245911 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2"] Nov 02 01:27:55.302396 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:55.276787 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b6ec07\"/\"distinct-w4-b6ec07-kserve-self-signed-certs\"" Nov 02 01:27:55.302396 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:55.276899 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b6ec07\"/\"openshift-service-ca.crt\"" Nov 02 01:27:55.302396 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:55.276997 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b6ec07\"/\"kube-root-ca.crt\"" Nov 02 01:27:55.302396 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:27:55.294620 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod165f361d_6b06_4c6c_8598_27c4b522b2e2.slice/crio-81dcfffc8047cdcf530580f4d46323763bc51aff3f133ec1de8aeb6f15a2c248 WatchSource:0}: Error finding container 81dcfffc8047cdcf530580f4d46323763bc51aff3f133ec1de8aeb6f15a2c248: Status 404 returned error can't find the container with id 81dcfffc8047cdcf530580f4d46323763bc51aff3f133ec1de8aeb6f15a2c248 Nov 02 01:27:55.333752 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:55.310851 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:27:55.333752 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:55.310941 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:27:55.333752 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:55.311009 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:27:55.333752 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:55.329664 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/353a8a41-21ee-4470-861c-b53833beb106-tmp-dir\") pod \"distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7\" (UID: \"353a8a41-21ee-4470-861c-b53833beb106\") " pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" Nov 02 01:27:55.367662 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:55.362929 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/353a8a41-21ee-4470-861c-b53833beb106-tmp-dir\") pod \"distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7\" (UID: \"353a8a41-21ee-4470-861c-b53833beb106\") " pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" Nov 02 01:27:55.397970 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:55.391458 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm2p9\" (UniqueName: \"kubernetes.io/projected/353a8a41-21ee-4470-861c-b53833beb106-kube-api-access-zm2p9\") pod \"distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7\" (UID: \"353a8a41-21ee-4470-861c-b53833beb106\") " pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" Nov 02 01:27:55.397970 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:55.391678 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/353a8a41-21ee-4470-861c-b53833beb106-home\") pod \"distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7\" (UID: \"353a8a41-21ee-4470-861c-b53833beb106\") " pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" Nov 02 01:27:55.397970 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:55.391742 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/353a8a41-21ee-4470-861c-b53833beb106-tls-certs\") pod \"distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7\" (UID: \"353a8a41-21ee-4470-861c-b53833beb106\") " pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" Nov 02 01:27:55.397970 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:55.391775 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/353a8a41-21ee-4470-861c-b53833beb106-model-cache\") pod \"distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7\" (UID: \"353a8a41-21ee-4470-861c-b53833beb106\") " pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" Nov 02 01:27:55.397970 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:55.391858 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/353a8a41-21ee-4470-861c-b53833beb106-dshm\") pod \"distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7\" (UID: \"353a8a41-21ee-4470-861c-b53833beb106\") " pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" Nov 02 01:27:55.397970 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:55.393856 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/353a8a41-21ee-4470-861c-b53833beb106-home\") pod \"distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7\" (UID: \"353a8a41-21ee-4470-861c-b53833beb106\") " pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" Nov 02 01:27:55.397970 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:55.394993 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/353a8a41-21ee-4470-861c-b53833beb106-model-cache\") pod \"distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7\" (UID: \"353a8a41-21ee-4470-861c-b53833beb106\") " pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" Nov 02 01:27:55.414488 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:55.401594 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/353a8a41-21ee-4470-861c-b53833beb106-dshm\") pod \"distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7\" (UID: \"353a8a41-21ee-4470-861c-b53833beb106\") " pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" Nov 02 01:27:55.414488 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:55.410052 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/353a8a41-21ee-4470-861c-b53833beb106-tls-certs\") pod \"distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7\" (UID: \"353a8a41-21ee-4470-861c-b53833beb106\") " pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" Nov 02 01:27:55.414789 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:55.414728 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm2p9\" (UniqueName: \"kubernetes.io/projected/353a8a41-21ee-4470-861c-b53833beb106-kube-api-access-zm2p9\") pod \"distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7\" (UID: \"353a8a41-21ee-4470-861c-b53833beb106\") " pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" Nov 02 01:27:55.415523 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:55.415503 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" Nov 02 01:27:55.603761 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:55.603728 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv"] Nov 02 01:27:55.698954 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:55.698903 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:27:55.701075 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:55.701048 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:27:55.701271 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:55.701244 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:27:55.737123 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:55.737090 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv" event={"ID":"819ae12d-e493-42d0-a549-811fb5fe7499","Type":"ContainerStarted","Data":"fedfa286623e6c7075cecc115795fbdbd6d127ce9bd6417cd3d85d588d75aabb"} Nov 02 01:27:55.739147 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:55.739119 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2" event={"ID":"165f361d-6b06-4c6c-8598-27c4b522b2e2","Type":"ContainerStarted","Data":"a931dca1e3a0c74d664fbab0dc4d3377ef5d222942da09c5888c3ba48bab3be2"} Nov 02 01:27:55.739839 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:55.739801 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2" event={"ID":"165f361d-6b06-4c6c-8598-27c4b522b2e2","Type":"ContainerStarted","Data":"81dcfffc8047cdcf530580f4d46323763bc51aff3f133ec1de8aeb6f15a2c248"} Nov 02 01:27:55.786950 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:55.786888 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2" podStartSLOduration=1.786865938 podStartE2EDuration="1.786865938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:27:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:27:55.772794577 +0000 UTC m=+780.393799801" watchObservedRunningTime="2026-11-02 01:27:55.786865938 +0000 UTC m=+780.407871153" Nov 02 01:27:55.810766 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:55.810734 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7"] Nov 02 01:27:56.724728 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:56.724695 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2" Nov 02 01:27:56.727831 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:56.727804 4427 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 02 01:27:56.728096 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:56.728070 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2" podUID="165f361d-6b06-4c6c-8598-27c4b522b2e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 02 01:27:56.747848 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:56.747786 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv" event={"ID":"819ae12d-e493-42d0-a549-811fb5fe7499","Type":"ContainerStarted","Data":"86bcb723f3ac09eb08da6a53144ab90378f17002034572135c475f435bf13c0c"} Nov 02 01:27:56.749415 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:56.749351 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" event={"ID":"353a8a41-21ee-4470-861c-b53833beb106","Type":"ContainerStarted","Data":"0901206f61d6087705a79fd16a87b60f61e8303a71c2cad4b8c40990a2f1a653"} Nov 02 01:27:56.749415 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:56.749395 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" event={"ID":"353a8a41-21ee-4470-861c-b53833beb106","Type":"ContainerStarted","Data":"8af4b67ddc044e34300cb434ec421ebb8060a5136a1b8f478421b3defd7f9167"} Nov 02 01:27:56.778083 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:56.778010 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv" podStartSLOduration=2.777987304 podStartE2EDuration="2.777987304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:27:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:27:56.771836275 +0000 UTC m=+781.392841512" watchObservedRunningTime="2026-11-02 01:27:56.777987304 +0000 UTC m=+781.398992517" Nov 02 01:27:56.786823 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:56.785239 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" Nov 02 01:27:56.835315 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:56.835289 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" Nov 02 01:27:56.854580 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:56.854529 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" podStartSLOduration=1.8545109 podStartE2EDuration="1.8545109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:27:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:27:56.804566388 +0000 UTC m=+781.425571602" watchObservedRunningTime="2026-11-02 01:27:56.8545109 +0000 UTC m=+781.475516102" Nov 02 01:27:57.416803 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:57.416754 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" Nov 02 01:27:57.723760 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:57.723676 4427 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 02 01:27:57.723760 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:57.723732 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2" podUID="165f361d-6b06-4c6c-8598-27c4b522b2e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 02 01:27:57.724079 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:57.724057 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv" Nov 02 01:27:57.725396 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:57.725359 4427 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 02 01:27:57.725692 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:57.725417 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv" podUID="819ae12d-e493-42d0-a549-811fb5fe7499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 02 01:27:57.821265 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:57.821221 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" Nov 02 01:27:58.724214 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:58.724163 4427 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 02 01:27:58.724425 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:58.724225 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2" podUID="165f361d-6b06-4c6c-8598-27c4b522b2e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 02 01:27:58.724425 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:58.724209 4427 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 02 01:27:58.724425 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:58.724272 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv" podUID="819ae12d-e493-42d0-a549-811fb5fe7499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 02 01:27:59.724421 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:59.724061 4427 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 02 01:27:59.724421 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:59.724131 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv" podUID="819ae12d-e493-42d0-a549-811fb5fe7499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 02 01:27:59.725020 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:59.724521 4427 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 02 01:27:59.725020 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:27:59.724566 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2" podUID="165f361d-6b06-4c6c-8598-27c4b522b2e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 02 01:28:00.725779 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:00.725736 4427 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 02 01:28:00.726329 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:00.725797 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv" podUID="819ae12d-e493-42d0-a549-811fb5fe7499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 02 01:28:00.726514 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:00.726490 4427 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 02 01:28:00.726584 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:00.726532 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2" podUID="165f361d-6b06-4c6c-8598-27c4b522b2e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 02 01:28:01.651398 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:28:01.649412 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:28:01.726788 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:01.726746 4427 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 02 01:28:01.727287 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:01.726809 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2" podUID="165f361d-6b06-4c6c-8598-27c4b522b2e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 02 01:28:01.727400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:01.727357 4427 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 02 01:28:01.727466 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:01.727411 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv" podUID="819ae12d-e493-42d0-a549-811fb5fe7499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 02 01:28:02.724836 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:02.724289 4427 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 02 01:28:02.724836 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:02.724362 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2" podUID="165f361d-6b06-4c6c-8598-27c4b522b2e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 02 01:28:02.856219 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:02.856184 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv" Nov 02 01:28:02.857437 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:02.857362 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv" Nov 02 01:28:02.858869 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:02.858841 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv" Nov 02 01:28:03.729158 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:03.729131 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2" Nov 02 01:28:03.729585 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:03.729563 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2" Nov 02 01:28:03.730359 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:03.730334 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2" Nov 02 01:28:05.400783 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:05.400742 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk"] Nov 02 01:28:05.405001 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:05.404974 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" Nov 02 01:28:05.416675 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:05.416132 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk"] Nov 02 01:28:05.416675 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:05.416415 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" Nov 02 01:28:05.416675 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:05.416631 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-123986\"/\"distinct-w5-123986-kserve-self-signed-certs\"" Nov 02 01:28:05.433723 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:05.433691 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" Nov 02 01:28:05.496124 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:05.496085 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f67c954a-ce5b-41d9-8d99-7d126bd89d59-tls-certs\") pod \"distinct-w5-123986-kserve-79b985685c-xvlxk\" (UID: \"f67c954a-ce5b-41d9-8d99-7d126bd89d59\") " pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" Nov 02 01:28:05.496124 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:05.496120 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f67c954a-ce5b-41d9-8d99-7d126bd89d59-home\") pod \"distinct-w5-123986-kserve-79b985685c-xvlxk\" (UID: \"f67c954a-ce5b-41d9-8d99-7d126bd89d59\") " pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" Nov 02 01:28:05.496408 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:05.496306 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f67c954a-ce5b-41d9-8d99-7d126bd89d59-dshm\") pod \"distinct-w5-123986-kserve-79b985685c-xvlxk\" (UID: \"f67c954a-ce5b-41d9-8d99-7d126bd89d59\") " pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" Nov 02 01:28:05.496482 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:05.496463 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwh7v\" (UniqueName: \"kubernetes.io/projected/f67c954a-ce5b-41d9-8d99-7d126bd89d59-kube-api-access-pwh7v\") pod \"distinct-w5-123986-kserve-79b985685c-xvlxk\" (UID: \"f67c954a-ce5b-41d9-8d99-7d126bd89d59\") " pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" Nov 02 01:28:05.496540 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:05.496498 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f67c954a-ce5b-41d9-8d99-7d126bd89d59-model-cache\") pod \"distinct-w5-123986-kserve-79b985685c-xvlxk\" (UID: \"f67c954a-ce5b-41d9-8d99-7d126bd89d59\") " pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" Nov 02 01:28:05.496631 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:05.496615 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f67c954a-ce5b-41d9-8d99-7d126bd89d59-tmp-dir\") pod \"distinct-w5-123986-kserve-79b985685c-xvlxk\" (UID: \"f67c954a-ce5b-41d9-8d99-7d126bd89d59\") " pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" Nov 02 01:28:05.607400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:05.603208 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwh7v\" (UniqueName: \"kubernetes.io/projected/f67c954a-ce5b-41d9-8d99-7d126bd89d59-kube-api-access-pwh7v\") pod \"distinct-w5-123986-kserve-79b985685c-xvlxk\" (UID: \"f67c954a-ce5b-41d9-8d99-7d126bd89d59\") " pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" Nov 02 01:28:05.607400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:05.603261 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f67c954a-ce5b-41d9-8d99-7d126bd89d59-model-cache\") pod \"distinct-w5-123986-kserve-79b985685c-xvlxk\" (UID: \"f67c954a-ce5b-41d9-8d99-7d126bd89d59\") " pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" Nov 02 01:28:05.607400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:05.603350 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f67c954a-ce5b-41d9-8d99-7d126bd89d59-tmp-dir\") pod \"distinct-w5-123986-kserve-79b985685c-xvlxk\" (UID: \"f67c954a-ce5b-41d9-8d99-7d126bd89d59\") " pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" Nov 02 01:28:05.607400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:05.605032 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f67c954a-ce5b-41d9-8d99-7d126bd89d59-tls-certs\") pod \"distinct-w5-123986-kserve-79b985685c-xvlxk\" (UID: \"f67c954a-ce5b-41d9-8d99-7d126bd89d59\") " pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" Nov 02 01:28:05.607400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:05.605073 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f67c954a-ce5b-41d9-8d99-7d126bd89d59-home\") pod \"distinct-w5-123986-kserve-79b985685c-xvlxk\" (UID: \"f67c954a-ce5b-41d9-8d99-7d126bd89d59\") " pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" Nov 02 01:28:05.607400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:05.605172 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f67c954a-ce5b-41d9-8d99-7d126bd89d59-dshm\") pod \"distinct-w5-123986-kserve-79b985685c-xvlxk\" (UID: \"f67c954a-ce5b-41d9-8d99-7d126bd89d59\") " pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" Nov 02 01:28:05.607400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:05.605996 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f67c954a-ce5b-41d9-8d99-7d126bd89d59-tmp-dir\") pod \"distinct-w5-123986-kserve-79b985685c-xvlxk\" (UID: \"f67c954a-ce5b-41d9-8d99-7d126bd89d59\") " pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" Nov 02 01:28:05.632785 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:05.632751 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f67c954a-ce5b-41d9-8d99-7d126bd89d59-model-cache\") pod \"distinct-w5-123986-kserve-79b985685c-xvlxk\" (UID: \"f67c954a-ce5b-41d9-8d99-7d126bd89d59\") " pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" Nov 02 01:28:05.633584 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:05.633559 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f67c954a-ce5b-41d9-8d99-7d126bd89d59-dshm\") pod \"distinct-w5-123986-kserve-79b985685c-xvlxk\" (UID: \"f67c954a-ce5b-41d9-8d99-7d126bd89d59\") " pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" Nov 02 01:28:05.633771 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:05.633745 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f67c954a-ce5b-41d9-8d99-7d126bd89d59-home\") pod \"distinct-w5-123986-kserve-79b985685c-xvlxk\" (UID: \"f67c954a-ce5b-41d9-8d99-7d126bd89d59\") " pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" Nov 02 01:28:05.655482 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:05.655147 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f67c954a-ce5b-41d9-8d99-7d126bd89d59-tls-certs\") pod \"distinct-w5-123986-kserve-79b985685c-xvlxk\" (UID: \"f67c954a-ce5b-41d9-8d99-7d126bd89d59\") " pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" Nov 02 01:28:05.655769 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:05.655746 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwh7v\" (UniqueName: \"kubernetes.io/projected/f67c954a-ce5b-41d9-8d99-7d126bd89d59-kube-api-access-pwh7v\") pod \"distinct-w5-123986-kserve-79b985685c-xvlxk\" (UID: \"f67c954a-ce5b-41d9-8d99-7d126bd89d59\") " pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" Nov 02 01:28:05.721265 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:05.721229 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" Nov 02 01:28:05.881128 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:05.881100 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" Nov 02 01:28:06.013036 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:06.013009 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk"] Nov 02 01:28:06.014846 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:28:06.014813 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf67c954a_ce5b_41d9_8d99_7d126bd89d59.slice/crio-f4141605995397b676fb7e47daec7f28e443b6eb9fbc6b65ac2cdf652e0b31df WatchSource:0}: Error finding container f4141605995397b676fb7e47daec7f28e443b6eb9fbc6b65ac2cdf652e0b31df: Status 404 returned error can't find the container with id f4141605995397b676fb7e47daec7f28e443b6eb9fbc6b65ac2cdf652e0b31df Nov 02 01:28:06.876204 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:06.876165 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" event={"ID":"f67c954a-ce5b-41d9-8d99-7d126bd89d59","Type":"ContainerStarted","Data":"1177e0754c41f78470f9ec6480093861574c18bb76394d04f56bae230a51ee03"} Nov 02 01:28:06.876204 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:06.876207 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" event={"ID":"f67c954a-ce5b-41d9-8d99-7d126bd89d59","Type":"ContainerStarted","Data":"f4141605995397b676fb7e47daec7f28e443b6eb9fbc6b65ac2cdf652e0b31df"} Nov 02 01:28:06.899152 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:06.899099 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" podStartSLOduration=1.899076548 podStartE2EDuration="1.899076548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:28:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:28:06.89740087 +0000 UTC m=+791.518406084" watchObservedRunningTime="2026-11-02 01:28:06.899076548 +0000 UTC m=+791.520081759" Nov 02 01:28:07.722409 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:07.722353 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" Nov 02 01:28:14.070811 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:14.070769 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks"] Nov 02 01:28:14.075690 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:14.075664 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" Nov 02 01:28:14.079247 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:14.079215 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks"] Nov 02 01:28:14.080588 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:14.080560 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5687bc\"/\"distinct-w1-5687bc-kserve-self-signed-certs\"" Nov 02 01:28:14.115525 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:14.115484 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/366c78ee-4aaf-4b7c-84b4-17028538cb1f-tmp-dir\") pod \"distinct-w1-5687bc-kserve-55887d7865-456ks\" (UID: \"366c78ee-4aaf-4b7c-84b4-17028538cb1f\") " pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" Nov 02 01:28:14.115708 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:14.115539 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/366c78ee-4aaf-4b7c-84b4-17028538cb1f-home\") pod \"distinct-w1-5687bc-kserve-55887d7865-456ks\" (UID: \"366c78ee-4aaf-4b7c-84b4-17028538cb1f\") " pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" Nov 02 01:28:14.115708 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:14.115663 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/366c78ee-4aaf-4b7c-84b4-17028538cb1f-model-cache\") pod \"distinct-w1-5687bc-kserve-55887d7865-456ks\" (UID: \"366c78ee-4aaf-4b7c-84b4-17028538cb1f\") " pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" Nov 02 01:28:14.115808 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:14.115706 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmgdv\" (UniqueName: \"kubernetes.io/projected/366c78ee-4aaf-4b7c-84b4-17028538cb1f-kube-api-access-kmgdv\") pod \"distinct-w1-5687bc-kserve-55887d7865-456ks\" (UID: \"366c78ee-4aaf-4b7c-84b4-17028538cb1f\") " pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" Nov 02 01:28:14.115946 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:14.115928 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/366c78ee-4aaf-4b7c-84b4-17028538cb1f-tls-certs\") pod \"distinct-w1-5687bc-kserve-55887d7865-456ks\" (UID: \"366c78ee-4aaf-4b7c-84b4-17028538cb1f\") " pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" Nov 02 01:28:14.116060 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:14.116041 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/366c78ee-4aaf-4b7c-84b4-17028538cb1f-dshm\") pod \"distinct-w1-5687bc-kserve-55887d7865-456ks\" (UID: \"366c78ee-4aaf-4b7c-84b4-17028538cb1f\") " pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" Nov 02 01:28:14.223405 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:14.221013 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/366c78ee-4aaf-4b7c-84b4-17028538cb1f-tls-certs\") pod \"distinct-w1-5687bc-kserve-55887d7865-456ks\" (UID: \"366c78ee-4aaf-4b7c-84b4-17028538cb1f\") " pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" Nov 02 01:28:14.223405 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:14.221090 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/366c78ee-4aaf-4b7c-84b4-17028538cb1f-dshm\") pod \"distinct-w1-5687bc-kserve-55887d7865-456ks\" (UID: \"366c78ee-4aaf-4b7c-84b4-17028538cb1f\") " pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" Nov 02 01:28:14.223405 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:14.221347 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/366c78ee-4aaf-4b7c-84b4-17028538cb1f-tmp-dir\") pod \"distinct-w1-5687bc-kserve-55887d7865-456ks\" (UID: \"366c78ee-4aaf-4b7c-84b4-17028538cb1f\") " pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" Nov 02 01:28:14.223405 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:14.221402 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/366c78ee-4aaf-4b7c-84b4-17028538cb1f-home\") pod \"distinct-w1-5687bc-kserve-55887d7865-456ks\" (UID: \"366c78ee-4aaf-4b7c-84b4-17028538cb1f\") " pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" Nov 02 01:28:14.223405 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:14.221530 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/366c78ee-4aaf-4b7c-84b4-17028538cb1f-model-cache\") pod \"distinct-w1-5687bc-kserve-55887d7865-456ks\" (UID: \"366c78ee-4aaf-4b7c-84b4-17028538cb1f\") " pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" Nov 02 01:28:14.223405 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:14.221565 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmgdv\" (UniqueName: \"kubernetes.io/projected/366c78ee-4aaf-4b7c-84b4-17028538cb1f-kube-api-access-kmgdv\") pod \"distinct-w1-5687bc-kserve-55887d7865-456ks\" (UID: \"366c78ee-4aaf-4b7c-84b4-17028538cb1f\") " pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" Nov 02 01:28:14.223405 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:14.222905 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/366c78ee-4aaf-4b7c-84b4-17028538cb1f-home\") pod \"distinct-w1-5687bc-kserve-55887d7865-456ks\" (UID: \"366c78ee-4aaf-4b7c-84b4-17028538cb1f\") " pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" Nov 02 01:28:14.223405 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:14.223029 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/366c78ee-4aaf-4b7c-84b4-17028538cb1f-model-cache\") pod \"distinct-w1-5687bc-kserve-55887d7865-456ks\" (UID: \"366c78ee-4aaf-4b7c-84b4-17028538cb1f\") " pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" Nov 02 01:28:14.223903 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:14.223591 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/366c78ee-4aaf-4b7c-84b4-17028538cb1f-tmp-dir\") pod \"distinct-w1-5687bc-kserve-55887d7865-456ks\" (UID: \"366c78ee-4aaf-4b7c-84b4-17028538cb1f\") " pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" Nov 02 01:28:14.241404 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:14.239198 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/366c78ee-4aaf-4b7c-84b4-17028538cb1f-tls-certs\") pod \"distinct-w1-5687bc-kserve-55887d7865-456ks\" (UID: \"366c78ee-4aaf-4b7c-84b4-17028538cb1f\") " pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" Nov 02 01:28:14.241404 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:14.240126 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/366c78ee-4aaf-4b7c-84b4-17028538cb1f-dshm\") pod \"distinct-w1-5687bc-kserve-55887d7865-456ks\" (UID: \"366c78ee-4aaf-4b7c-84b4-17028538cb1f\") " pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" Nov 02 01:28:14.241404 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:14.240855 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmgdv\" (UniqueName: \"kubernetes.io/projected/366c78ee-4aaf-4b7c-84b4-17028538cb1f-kube-api-access-kmgdv\") pod \"distinct-w1-5687bc-kserve-55887d7865-456ks\" (UID: \"366c78ee-4aaf-4b7c-84b4-17028538cb1f\") " pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" Nov 02 01:28:14.393166 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:14.393118 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" Nov 02 01:28:14.678177 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:14.678140 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks"] Nov 02 01:28:14.679300 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:28:14.679273 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod366c78ee_4aaf_4b7c_84b4_17028538cb1f.slice/crio-bd3176f75d8edb89015f89c9b01237c740c67bb1bff83984a205d38b0419890a WatchSource:0}: Error finding container bd3176f75d8edb89015f89c9b01237c740c67bb1bff83984a205d38b0419890a: Status 404 returned error can't find the container with id bd3176f75d8edb89015f89c9b01237c740c67bb1bff83984a205d38b0419890a Nov 02 01:28:14.987992 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:14.987905 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" event={"ID":"366c78ee-4aaf-4b7c-84b4-17028538cb1f","Type":"ContainerStarted","Data":"7487fc8f9274de406bc39360f24319f7e915f6b1e443d16558494a87ccada363"} Nov 02 01:28:14.987992 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:14.987950 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" event={"ID":"366c78ee-4aaf-4b7c-84b4-17028538cb1f","Type":"ContainerStarted","Data":"bd3176f75d8edb89015f89c9b01237c740c67bb1bff83984a205d38b0419890a"} Nov 02 01:28:15.009197 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:15.009144 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" podStartSLOduration=1.009130937 podStartE2EDuration="1.009130937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:28:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:28:15.006445311 +0000 UTC m=+799.627450536" watchObservedRunningTime="2026-11-02 01:28:15.009130937 +0000 UTC m=+799.630136148" Nov 02 01:28:15.393884 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:15.393852 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" Nov 02 01:28:15.721682 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:15.721578 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" Nov 02 01:28:15.734563 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:15.734536 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" Nov 02 01:28:16.019153 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:16.019054 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" Nov 02 01:28:18.013535 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:18.013485 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" podUID="f67c954a-ce5b-41d9-8d99-7d126bd89d59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:28:18.690399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:18.687466 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" podUID="7085c0ab-7ee3-4665-abad-2977c25f68ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:28:18.877397 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:18.874514 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" podUID="353a8a41-21ee-4470-861c-b53833beb106" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:28:24.398954 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:24.398917 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" Nov 02 01:28:24.415874 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:24.415839 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" Nov 02 01:28:25.086737 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:25.086706 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" Nov 02 01:28:26.027952 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:26.027913 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv"] Nov 02 01:28:26.049786 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:26.049754 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" Nov 02 01:28:26.050647 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:26.050621 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv"] Nov 02 01:28:26.057185 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:26.057148 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b6ec07\"/\"distinct-2-w4-b6ec07-kserve-self-signed-certs\"" Nov 02 01:28:26.175397 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:26.170591 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9-home\") pod \"distinct-2-w4-b6ec07-kserve-6575469d47-l7crv\" (UID: \"cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9\") " pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" Nov 02 01:28:26.175397 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:26.170673 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9-dshm\") pod \"distinct-2-w4-b6ec07-kserve-6575469d47-l7crv\" (UID: \"cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9\") " pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" Nov 02 01:28:26.175397 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:26.170749 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9-model-cache\") pod \"distinct-2-w4-b6ec07-kserve-6575469d47-l7crv\" (UID: \"cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9\") " pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" Nov 02 01:28:26.175397 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:26.170816 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9-tls-certs\") pod \"distinct-2-w4-b6ec07-kserve-6575469d47-l7crv\" (UID: \"cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9\") " pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" Nov 02 01:28:26.175397 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:26.170857 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9-tmp-dir\") pod \"distinct-2-w4-b6ec07-kserve-6575469d47-l7crv\" (UID: \"cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9\") " pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" Nov 02 01:28:26.175397 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:26.170995 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm5kp\" (UniqueName: \"kubernetes.io/projected/cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9-kube-api-access-rm5kp\") pod \"distinct-2-w4-b6ec07-kserve-6575469d47-l7crv\" (UID: \"cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9\") " pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" Nov 02 01:28:26.271953 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:26.271922 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9-home\") pod \"distinct-2-w4-b6ec07-kserve-6575469d47-l7crv\" (UID: \"cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9\") " pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" Nov 02 01:28:26.272164 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:26.271976 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9-dshm\") pod \"distinct-2-w4-b6ec07-kserve-6575469d47-l7crv\" (UID: \"cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9\") " pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" Nov 02 01:28:26.272164 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:26.272032 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9-model-cache\") pod \"distinct-2-w4-b6ec07-kserve-6575469d47-l7crv\" (UID: \"cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9\") " pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" Nov 02 01:28:26.272164 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:26.272068 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9-tls-certs\") pod \"distinct-2-w4-b6ec07-kserve-6575469d47-l7crv\" (UID: \"cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9\") " pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" Nov 02 01:28:26.272164 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:26.272106 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9-tmp-dir\") pod \"distinct-2-w4-b6ec07-kserve-6575469d47-l7crv\" (UID: \"cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9\") " pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" Nov 02 01:28:26.272414 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:26.272234 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm5kp\" (UniqueName: \"kubernetes.io/projected/cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9-kube-api-access-rm5kp\") pod \"distinct-2-w4-b6ec07-kserve-6575469d47-l7crv\" (UID: \"cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9\") " pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" Nov 02 01:28:26.272480 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:26.272410 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9-home\") pod \"distinct-2-w4-b6ec07-kserve-6575469d47-l7crv\" (UID: \"cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9\") " pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" Nov 02 01:28:26.272741 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:26.272719 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9-model-cache\") pod \"distinct-2-w4-b6ec07-kserve-6575469d47-l7crv\" (UID: \"cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9\") " pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" Nov 02 01:28:26.272894 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:26.272847 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9-tmp-dir\") pod \"distinct-2-w4-b6ec07-kserve-6575469d47-l7crv\" (UID: \"cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9\") " pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" Nov 02 01:28:26.275169 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:26.275140 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9-dshm\") pod \"distinct-2-w4-b6ec07-kserve-6575469d47-l7crv\" (UID: \"cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9\") " pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" Nov 02 01:28:26.275479 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:26.275457 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9-tls-certs\") pod \"distinct-2-w4-b6ec07-kserve-6575469d47-l7crv\" (UID: \"cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9\") " pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" Nov 02 01:28:26.284707 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:26.284640 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm5kp\" (UniqueName: \"kubernetes.io/projected/cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9-kube-api-access-rm5kp\") pod \"distinct-2-w4-b6ec07-kserve-6575469d47-l7crv\" (UID: \"cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9\") " pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" Nov 02 01:28:26.381396 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:26.381341 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" Nov 02 01:28:26.671007 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:26.670942 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv"] Nov 02 01:28:27.094875 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:27.094776 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" event={"ID":"cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9","Type":"ContainerStarted","Data":"6b6477df2cccdc50f6e7994f7b6adf0b04d30466c18186b06337a11b4401b868"} Nov 02 01:28:27.094875 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:27.094820 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" event={"ID":"cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9","Type":"ContainerStarted","Data":"0e4d0b87c85e2de7e0a1d7b49a6e8709e9ffd52c251cae0cca36c868fa3c298d"} Nov 02 01:28:27.382415 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:27.382369 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" Nov 02 01:28:31.582959 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:28:31.582921 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:28:36.382586 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:36.382545 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" Nov 02 01:28:36.396687 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:36.396657 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" Nov 02 01:28:36.421906 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:36.421843 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" podStartSLOduration=10.421824312 podStartE2EDuration="10.421824312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:28:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:28:27.113596756 +0000 UTC m=+811.734601968" watchObservedRunningTime="2026-11-02 01:28:36.421824312 +0000 UTC m=+821.042829525" Nov 02 01:28:37.241461 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:37.240260 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" Nov 02 01:28:39.141249 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.141212 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm"] Nov 02 01:28:39.146896 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.146872 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm" Nov 02 01:28:39.163020 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.162989 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm"] Nov 02 01:28:39.190176 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.190145 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/119ffe15-7bac-418f-a288-52e1e61203ed-istiod-ca-cert\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm\" (UID: \"119ffe15-7bac-418f-a288-52e1e61203ed\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm" Nov 02 01:28:39.190363 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.190187 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/119ffe15-7bac-418f-a288-52e1e61203ed-istio-envoy\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm\" (UID: \"119ffe15-7bac-418f-a288-52e1e61203ed\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm" Nov 02 01:28:39.190363 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.190215 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/119ffe15-7bac-418f-a288-52e1e61203ed-workload-socket\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm\" (UID: \"119ffe15-7bac-418f-a288-52e1e61203ed\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm" Nov 02 01:28:39.190516 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.190343 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/119ffe15-7bac-418f-a288-52e1e61203ed-credential-socket\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm\" (UID: \"119ffe15-7bac-418f-a288-52e1e61203ed\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm" Nov 02 01:28:39.190516 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.190484 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sm92\" (UniqueName: \"kubernetes.io/projected/119ffe15-7bac-418f-a288-52e1e61203ed-kube-api-access-4sm92\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm\" (UID: \"119ffe15-7bac-418f-a288-52e1e61203ed\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm" Nov 02 01:28:39.190610 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.190533 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/119ffe15-7bac-418f-a288-52e1e61203ed-workload-certs\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm\" (UID: \"119ffe15-7bac-418f-a288-52e1e61203ed\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm" Nov 02 01:28:39.190702 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.190675 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/119ffe15-7bac-418f-a288-52e1e61203ed-istio-token\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm\" (UID: \"119ffe15-7bac-418f-a288-52e1e61203ed\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm" Nov 02 01:28:39.190755 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.190732 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/119ffe15-7bac-418f-a288-52e1e61203ed-istio-podinfo\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm\" (UID: \"119ffe15-7bac-418f-a288-52e1e61203ed\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm" Nov 02 01:28:39.190829 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.190812 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/119ffe15-7bac-418f-a288-52e1e61203ed-istio-data\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm\" (UID: \"119ffe15-7bac-418f-a288-52e1e61203ed\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm" Nov 02 01:28:39.291859 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.291825 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/119ffe15-7bac-418f-a288-52e1e61203ed-istiod-ca-cert\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm\" (UID: \"119ffe15-7bac-418f-a288-52e1e61203ed\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm" Nov 02 01:28:39.292072 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.291869 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/119ffe15-7bac-418f-a288-52e1e61203ed-istio-envoy\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm\" (UID: \"119ffe15-7bac-418f-a288-52e1e61203ed\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm" Nov 02 01:28:39.292072 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.291893 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/119ffe15-7bac-418f-a288-52e1e61203ed-workload-socket\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm\" (UID: \"119ffe15-7bac-418f-a288-52e1e61203ed\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm" Nov 02 01:28:39.292072 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.291919 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/119ffe15-7bac-418f-a288-52e1e61203ed-credential-socket\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm\" (UID: \"119ffe15-7bac-418f-a288-52e1e61203ed\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm" Nov 02 01:28:39.292072 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.291969 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sm92\" (UniqueName: \"kubernetes.io/projected/119ffe15-7bac-418f-a288-52e1e61203ed-kube-api-access-4sm92\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm\" (UID: \"119ffe15-7bac-418f-a288-52e1e61203ed\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm" Nov 02 01:28:39.292072 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.291996 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/119ffe15-7bac-418f-a288-52e1e61203ed-workload-certs\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm\" (UID: \"119ffe15-7bac-418f-a288-52e1e61203ed\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm" Nov 02 01:28:39.292072 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.292064 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/119ffe15-7bac-418f-a288-52e1e61203ed-istio-token\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm\" (UID: \"119ffe15-7bac-418f-a288-52e1e61203ed\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm" Nov 02 01:28:39.292417 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.292111 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/119ffe15-7bac-418f-a288-52e1e61203ed-istio-podinfo\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm\" (UID: \"119ffe15-7bac-418f-a288-52e1e61203ed\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm" Nov 02 01:28:39.292417 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.292139 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/119ffe15-7bac-418f-a288-52e1e61203ed-istio-data\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm\" (UID: \"119ffe15-7bac-418f-a288-52e1e61203ed\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm" Nov 02 01:28:39.292923 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.292890 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/119ffe15-7bac-418f-a288-52e1e61203ed-workload-socket\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm\" (UID: \"119ffe15-7bac-418f-a288-52e1e61203ed\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm" Nov 02 01:28:39.293036 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.293010 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/119ffe15-7bac-418f-a288-52e1e61203ed-credential-socket\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm\" (UID: \"119ffe15-7bac-418f-a288-52e1e61203ed\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm" Nov 02 01:28:39.293209 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.293184 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/119ffe15-7bac-418f-a288-52e1e61203ed-istiod-ca-cert\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm\" (UID: \"119ffe15-7bac-418f-a288-52e1e61203ed\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm" Nov 02 01:28:39.293385 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.293284 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/119ffe15-7bac-418f-a288-52e1e61203ed-workload-certs\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm\" (UID: \"119ffe15-7bac-418f-a288-52e1e61203ed\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm" Nov 02 01:28:39.293527 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.293504 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/119ffe15-7bac-418f-a288-52e1e61203ed-istio-data\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm\" (UID: \"119ffe15-7bac-418f-a288-52e1e61203ed\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm" Nov 02 01:28:39.295139 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.295117 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/119ffe15-7bac-418f-a288-52e1e61203ed-istio-podinfo\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm\" (UID: \"119ffe15-7bac-418f-a288-52e1e61203ed\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm" Nov 02 01:28:39.295315 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.295293 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/119ffe15-7bac-418f-a288-52e1e61203ed-istio-envoy\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm\" (UID: \"119ffe15-7bac-418f-a288-52e1e61203ed\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm" Nov 02 01:28:39.304323 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.304299 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/119ffe15-7bac-418f-a288-52e1e61203ed-istio-token\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm\" (UID: \"119ffe15-7bac-418f-a288-52e1e61203ed\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm" Nov 02 01:28:39.304469 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.304359 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sm92\" (UniqueName: \"kubernetes.io/projected/119ffe15-7bac-418f-a288-52e1e61203ed-kube-api-access-4sm92\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm\" (UID: \"119ffe15-7bac-418f-a288-52e1e61203ed\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm" Nov 02 01:28:39.397565 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.397396 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28"] Nov 02 01:28:39.403553 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.403518 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28" Nov 02 01:28:39.422673 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.422638 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28"] Nov 02 01:28:39.462109 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.462074 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm" Nov 02 01:28:39.495114 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.495070 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1563efef-2ed3-4347-8399-4c9ca520db4b-credential-socket\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28\" (UID: \"1563efef-2ed3-4347-8399-4c9ca520db4b\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28" Nov 02 01:28:39.495291 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.495135 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1563efef-2ed3-4347-8399-4c9ca520db4b-istio-data\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28\" (UID: \"1563efef-2ed3-4347-8399-4c9ca520db4b\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28" Nov 02 01:28:39.495353 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.495310 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1563efef-2ed3-4347-8399-4c9ca520db4b-workload-socket\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28\" (UID: \"1563efef-2ed3-4347-8399-4c9ca520db4b\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28" Nov 02 01:28:39.495460 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.495443 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j5ff\" (UniqueName: \"kubernetes.io/projected/1563efef-2ed3-4347-8399-4c9ca520db4b-kube-api-access-7j5ff\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28\" (UID: \"1563efef-2ed3-4347-8399-4c9ca520db4b\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28" Nov 02 01:28:39.495504 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.495479 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1563efef-2ed3-4347-8399-4c9ca520db4b-workload-certs\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28\" (UID: \"1563efef-2ed3-4347-8399-4c9ca520db4b\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28" Nov 02 01:28:39.495599 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.495574 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1563efef-2ed3-4347-8399-4c9ca520db4b-istio-token\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28\" (UID: \"1563efef-2ed3-4347-8399-4c9ca520db4b\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28" Nov 02 01:28:39.495680 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.495662 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1563efef-2ed3-4347-8399-4c9ca520db4b-istio-envoy\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28\" (UID: \"1563efef-2ed3-4347-8399-4c9ca520db4b\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28" Nov 02 01:28:39.495717 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.495702 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1563efef-2ed3-4347-8399-4c9ca520db4b-istio-podinfo\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28\" (UID: \"1563efef-2ed3-4347-8399-4c9ca520db4b\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28" Nov 02 01:28:39.495880 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.495860 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1563efef-2ed3-4347-8399-4c9ca520db4b-istiod-ca-cert\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28\" (UID: \"1563efef-2ed3-4347-8399-4c9ca520db4b\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28" Nov 02 01:28:39.597574 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.597541 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1563efef-2ed3-4347-8399-4c9ca520db4b-workload-socket\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28\" (UID: \"1563efef-2ed3-4347-8399-4c9ca520db4b\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28" Nov 02 01:28:39.597768 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.597643 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j5ff\" (UniqueName: \"kubernetes.io/projected/1563efef-2ed3-4347-8399-4c9ca520db4b-kube-api-access-7j5ff\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28\" (UID: \"1563efef-2ed3-4347-8399-4c9ca520db4b\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28" Nov 02 01:28:39.597768 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.597673 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1563efef-2ed3-4347-8399-4c9ca520db4b-workload-certs\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28\" (UID: \"1563efef-2ed3-4347-8399-4c9ca520db4b\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28" Nov 02 01:28:39.597768 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.597706 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1563efef-2ed3-4347-8399-4c9ca520db4b-istio-token\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28\" (UID: \"1563efef-2ed3-4347-8399-4c9ca520db4b\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28" Nov 02 01:28:39.597768 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.597755 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1563efef-2ed3-4347-8399-4c9ca520db4b-istio-envoy\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28\" (UID: \"1563efef-2ed3-4347-8399-4c9ca520db4b\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28" Nov 02 01:28:39.598054 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.597796 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1563efef-2ed3-4347-8399-4c9ca520db4b-istio-podinfo\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28\" (UID: \"1563efef-2ed3-4347-8399-4c9ca520db4b\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28" Nov 02 01:28:39.598054 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.597939 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1563efef-2ed3-4347-8399-4c9ca520db4b-istiod-ca-cert\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28\" (UID: \"1563efef-2ed3-4347-8399-4c9ca520db4b\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28" Nov 02 01:28:39.598054 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.597991 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1563efef-2ed3-4347-8399-4c9ca520db4b-workload-socket\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28\" (UID: \"1563efef-2ed3-4347-8399-4c9ca520db4b\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28" Nov 02 01:28:39.598447 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.598417 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1563efef-2ed3-4347-8399-4c9ca520db4b-credential-socket\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28\" (UID: \"1563efef-2ed3-4347-8399-4c9ca520db4b\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28" Nov 02 01:28:39.598554 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.598480 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1563efef-2ed3-4347-8399-4c9ca520db4b-istio-data\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28\" (UID: \"1563efef-2ed3-4347-8399-4c9ca520db4b\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28" Nov 02 01:28:39.598656 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.598627 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1563efef-2ed3-4347-8399-4c9ca520db4b-istiod-ca-cert\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28\" (UID: \"1563efef-2ed3-4347-8399-4c9ca520db4b\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28" Nov 02 01:28:39.598864 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.598846 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1563efef-2ed3-4347-8399-4c9ca520db4b-workload-certs\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28\" (UID: \"1563efef-2ed3-4347-8399-4c9ca520db4b\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28" Nov 02 01:28:39.599105 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.599076 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1563efef-2ed3-4347-8399-4c9ca520db4b-credential-socket\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28\" (UID: \"1563efef-2ed3-4347-8399-4c9ca520db4b\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28" Nov 02 01:28:39.599332 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.599289 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1563efef-2ed3-4347-8399-4c9ca520db4b-istio-data\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28\" (UID: \"1563efef-2ed3-4347-8399-4c9ca520db4b\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28" Nov 02 01:28:39.600535 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.600492 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1563efef-2ed3-4347-8399-4c9ca520db4b-istio-envoy\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28\" (UID: \"1563efef-2ed3-4347-8399-4c9ca520db4b\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28" Nov 02 01:28:39.600859 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.600841 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1563efef-2ed3-4347-8399-4c9ca520db4b-istio-podinfo\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28\" (UID: \"1563efef-2ed3-4347-8399-4c9ca520db4b\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28" Nov 02 01:28:39.610839 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.610809 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j5ff\" (UniqueName: \"kubernetes.io/projected/1563efef-2ed3-4347-8399-4c9ca520db4b-kube-api-access-7j5ff\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28\" (UID: \"1563efef-2ed3-4347-8399-4c9ca520db4b\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28" Nov 02 01:28:39.611059 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.611042 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1563efef-2ed3-4347-8399-4c9ca520db4b-istio-token\") pod \"e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28\" (UID: \"1563efef-2ed3-4347-8399-4c9ca520db4b\") " pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28" Nov 02 01:28:39.731047 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.730972 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28" Nov 02 01:28:39.749309 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.749275 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm"] Nov 02 01:28:39.749664 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:28:39.749642 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod119ffe15_7bac_418f_a288_52e1e61203ed.slice/crio-adff0d4fce22cf6acc9a0f24fd77655f6ef15905e6d60d812bd2f5593395ecb1 WatchSource:0}: Error finding container adff0d4fce22cf6acc9a0f24fd77655f6ef15905e6d60d812bd2f5593395ecb1: Status 404 returned error can't find the container with id adff0d4fce22cf6acc9a0f24fd77655f6ef15905e6d60d812bd2f5593395ecb1 Nov 02 01:28:39.752466 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.752431 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:28:39.752569 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.752503 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:28:39.752569 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:39.752533 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:28:40.023547 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:28:40.023517 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1563efef_2ed3_4347_8399_4c9ca520db4b.slice/crio-96bc815453a44c30c2fb5298ead63b0ed81e0cea378b502f2096411ac31b7fe3 WatchSource:0}: Error finding container 96bc815453a44c30c2fb5298ead63b0ed81e0cea378b502f2096411ac31b7fe3: Status 404 returned error can't find the container with id 96bc815453a44c30c2fb5298ead63b0ed81e0cea378b502f2096411ac31b7fe3 Nov 02 01:28:40.024610 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:40.024577 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28"] Nov 02 01:28:40.031078 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:40.031050 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:28:40.031163 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:40.031119 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:28:40.031163 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:40.031151 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:28:40.244294 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:40.244254 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm" event={"ID":"119ffe15-7bac-418f-a288-52e1e61203ed","Type":"ContainerStarted","Data":"25e6f4621b2abdb4c7cbbc62df36ef508ae58c2545915107fc4a46092e22aaf0"} Nov 02 01:28:40.244825 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:40.244313 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm" event={"ID":"119ffe15-7bac-418f-a288-52e1e61203ed","Type":"ContainerStarted","Data":"adff0d4fce22cf6acc9a0f24fd77655f6ef15905e6d60d812bd2f5593395ecb1"} Nov 02 01:28:40.246402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:40.246345 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28" event={"ID":"1563efef-2ed3-4347-8399-4c9ca520db4b","Type":"ContainerStarted","Data":"1c33768a9daa419a56994957b1e1cf5c493d0dcd8cad607d9833533acbe3c5ea"} Nov 02 01:28:40.246513 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:40.246439 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28" event={"ID":"1563efef-2ed3-4347-8399-4c9ca520db4b","Type":"ContainerStarted","Data":"96bc815453a44c30c2fb5298ead63b0ed81e0cea378b502f2096411ac31b7fe3"} Nov 02 01:28:40.273562 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:40.273465 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm" podStartSLOduration=1.273450311 podStartE2EDuration="1.273450311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:28:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:28:40.269954989 +0000 UTC m=+824.890960224" watchObservedRunningTime="2026-11-02 01:28:40.273450311 +0000 UTC m=+824.894455588" Nov 02 01:28:40.300993 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:40.300925 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28" podStartSLOduration=1.3009020040000001 podStartE2EDuration="1.300902004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:28:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:28:40.296602459 +0000 UTC m=+824.917607694" watchObservedRunningTime="2026-11-02 01:28:40.300902004 +0000 UTC m=+824.921907216" Nov 02 01:28:40.462330 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:40.462286 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm" Nov 02 01:28:40.464047 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:40.464016 4427 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 01:28:40.464171 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:40.464071 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm" podUID="119ffe15-7bac-418f-a288-52e1e61203ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 01:28:40.731472 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:40.731441 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28" Nov 02 01:28:41.463903 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:41.463865 4427 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 01:28:41.464546 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:41.464515 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm" podUID="119ffe15-7bac-418f-a288-52e1e61203ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 01:28:41.733161 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:41.733078 4427 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 01:28:41.733161 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:41.733127 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28" podUID="1563efef-2ed3-4347-8399-4c9ca520db4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 01:28:42.470401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:42.463816 4427 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 01:28:42.470401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:42.463881 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm" podUID="119ffe15-7bac-418f-a288-52e1e61203ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 01:28:42.720677 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:42.720567 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr"] Nov 02 01:28:42.725495 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:42.725074 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" Nov 02 01:28:42.731498 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:42.731462 4427 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 01:28:42.731653 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:42.731515 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28" podUID="1563efef-2ed3-4347-8399-4c9ca520db4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 01:28:42.732507 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:42.732483 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr"] Nov 02 01:28:42.733387 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:42.733347 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5687bc\"/\"distinct-2-w1-5687bc-kserve-self-signed-certs\"" Nov 02 01:28:42.754827 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:42.754786 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z577g\" (UniqueName: \"kubernetes.io/projected/47a9069f-1ffa-4c83-be8b-5e62f1d3275a-kube-api-access-z577g\") pod \"distinct-2-w1-5687bc-kserve-5b98845f8-9blxr\" (UID: \"47a9069f-1ffa-4c83-be8b-5e62f1d3275a\") " pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" Nov 02 01:28:42.755588 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:42.755563 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47a9069f-1ffa-4c83-be8b-5e62f1d3275a-home\") pod \"distinct-2-w1-5687bc-kserve-5b98845f8-9blxr\" (UID: \"47a9069f-1ffa-4c83-be8b-5e62f1d3275a\") " pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" Nov 02 01:28:42.755812 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:42.755793 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47a9069f-1ffa-4c83-be8b-5e62f1d3275a-dshm\") pod \"distinct-2-w1-5687bc-kserve-5b98845f8-9blxr\" (UID: \"47a9069f-1ffa-4c83-be8b-5e62f1d3275a\") " pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" Nov 02 01:28:42.755941 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:42.755925 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47a9069f-1ffa-4c83-be8b-5e62f1d3275a-tmp-dir\") pod \"distinct-2-w1-5687bc-kserve-5b98845f8-9blxr\" (UID: \"47a9069f-1ffa-4c83-be8b-5e62f1d3275a\") " pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" Nov 02 01:28:42.756178 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:42.756159 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47a9069f-1ffa-4c83-be8b-5e62f1d3275a-model-cache\") pod \"distinct-2-w1-5687bc-kserve-5b98845f8-9blxr\" (UID: \"47a9069f-1ffa-4c83-be8b-5e62f1d3275a\") " pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" Nov 02 01:28:42.756328 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:42.756312 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47a9069f-1ffa-4c83-be8b-5e62f1d3275a-tls-certs\") pod \"distinct-2-w1-5687bc-kserve-5b98845f8-9blxr\" (UID: \"47a9069f-1ffa-4c83-be8b-5e62f1d3275a\") " pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" Nov 02 01:28:42.920513 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:42.916870 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47a9069f-1ffa-4c83-be8b-5e62f1d3275a-home\") pod \"distinct-2-w1-5687bc-kserve-5b98845f8-9blxr\" (UID: \"47a9069f-1ffa-4c83-be8b-5e62f1d3275a\") " pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" Nov 02 01:28:42.920513 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:42.916924 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47a9069f-1ffa-4c83-be8b-5e62f1d3275a-dshm\") pod \"distinct-2-w1-5687bc-kserve-5b98845f8-9blxr\" (UID: \"47a9069f-1ffa-4c83-be8b-5e62f1d3275a\") " pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" Nov 02 01:28:42.920513 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:42.916957 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47a9069f-1ffa-4c83-be8b-5e62f1d3275a-tmp-dir\") pod \"distinct-2-w1-5687bc-kserve-5b98845f8-9blxr\" (UID: \"47a9069f-1ffa-4c83-be8b-5e62f1d3275a\") " pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" Nov 02 01:28:42.920513 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:42.917089 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47a9069f-1ffa-4c83-be8b-5e62f1d3275a-model-cache\") pod \"distinct-2-w1-5687bc-kserve-5b98845f8-9blxr\" (UID: \"47a9069f-1ffa-4c83-be8b-5e62f1d3275a\") " pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" Nov 02 01:28:42.920513 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:42.917140 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47a9069f-1ffa-4c83-be8b-5e62f1d3275a-tls-certs\") pod \"distinct-2-w1-5687bc-kserve-5b98845f8-9blxr\" (UID: \"47a9069f-1ffa-4c83-be8b-5e62f1d3275a\") " pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" Nov 02 01:28:42.920513 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:42.917254 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z577g\" (UniqueName: \"kubernetes.io/projected/47a9069f-1ffa-4c83-be8b-5e62f1d3275a-kube-api-access-z577g\") pod \"distinct-2-w1-5687bc-kserve-5b98845f8-9blxr\" (UID: \"47a9069f-1ffa-4c83-be8b-5e62f1d3275a\") " pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" Nov 02 01:28:42.920513 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:42.918042 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47a9069f-1ffa-4c83-be8b-5e62f1d3275a-home\") pod \"distinct-2-w1-5687bc-kserve-5b98845f8-9blxr\" (UID: \"47a9069f-1ffa-4c83-be8b-5e62f1d3275a\") " pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" Nov 02 01:28:42.920513 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:42.918627 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47a9069f-1ffa-4c83-be8b-5e62f1d3275a-model-cache\") pod \"distinct-2-w1-5687bc-kserve-5b98845f8-9blxr\" (UID: \"47a9069f-1ffa-4c83-be8b-5e62f1d3275a\") " pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" Nov 02 01:28:42.920513 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:42.918876 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47a9069f-1ffa-4c83-be8b-5e62f1d3275a-tmp-dir\") pod \"distinct-2-w1-5687bc-kserve-5b98845f8-9blxr\" (UID: \"47a9069f-1ffa-4c83-be8b-5e62f1d3275a\") " pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" Nov 02 01:28:42.979505 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:42.961998 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z577g\" (UniqueName: \"kubernetes.io/projected/47a9069f-1ffa-4c83-be8b-5e62f1d3275a-kube-api-access-z577g\") pod \"distinct-2-w1-5687bc-kserve-5b98845f8-9blxr\" (UID: \"47a9069f-1ffa-4c83-be8b-5e62f1d3275a\") " pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" Nov 02 01:28:42.980367 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:42.980333 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47a9069f-1ffa-4c83-be8b-5e62f1d3275a-tls-certs\") pod \"distinct-2-w1-5687bc-kserve-5b98845f8-9blxr\" (UID: \"47a9069f-1ffa-4c83-be8b-5e62f1d3275a\") " pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" Nov 02 01:28:43.029412 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:43.028658 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47a9069f-1ffa-4c83-be8b-5e62f1d3275a-dshm\") pod \"distinct-2-w1-5687bc-kserve-5b98845f8-9blxr\" (UID: \"47a9069f-1ffa-4c83-be8b-5e62f1d3275a\") " pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" Nov 02 01:28:43.043920 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:43.043629 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" Nov 02 01:28:43.482401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:43.463798 4427 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 01:28:43.482401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:43.463858 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm" podUID="119ffe15-7bac-418f-a288-52e1e61203ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 01:28:43.538947 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:43.538918 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr"] Nov 02 01:28:43.734602 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:43.734566 4427 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 01:28:43.734809 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:43.734629 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28" podUID="1563efef-2ed3-4347-8399-4c9ca520db4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 01:28:44.350433 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:44.350399 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" event={"ID":"47a9069f-1ffa-4c83-be8b-5e62f1d3275a","Type":"ContainerStarted","Data":"ca04f4563d40cbea08860abbc1311b826618414b16f7cda510c5697f052dcd92"} Nov 02 01:28:44.350433 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:44.350440 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" event={"ID":"47a9069f-1ffa-4c83-be8b-5e62f1d3275a","Type":"ContainerStarted","Data":"864650fa9ce6f6a15ee4f4288d5e200e94fb3038d6cf4a9633fa1311d2b458b3"} Nov 02 01:28:44.483883 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:44.483842 4427 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 01:28:44.484520 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:44.484484 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm" podUID="119ffe15-7bac-418f-a288-52e1e61203ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 01:28:44.732300 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:44.732262 4427 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 01:28:44.732526 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:44.732318 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28" podUID="1563efef-2ed3-4347-8399-4c9ca520db4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 01:28:45.045063 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:45.044980 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" Nov 02 01:28:45.467404 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:45.463871 4427 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 01:28:45.467404 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:45.463939 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm" podUID="119ffe15-7bac-418f-a288-52e1e61203ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 01:28:45.731635 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:45.731551 4427 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 01:28:45.731996 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:45.731633 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28" podUID="1563efef-2ed3-4347-8399-4c9ca520db4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 01:28:46.463873 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:46.463835 4427 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 01:28:46.464171 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:46.464140 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm" podUID="119ffe15-7bac-418f-a288-52e1e61203ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 01:28:46.732459 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:46.732331 4427 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 01:28:46.732459 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:46.732433 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28" podUID="1563efef-2ed3-4347-8399-4c9ca520db4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 01:28:47.706498 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:47.706467 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm" Nov 02 01:28:47.707138 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:47.707112 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm" Nov 02 01:28:47.708025 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:47.708005 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm" Nov 02 01:28:47.737354 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:47.737290 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" podStartSLOduration=5.737273582 podStartE2EDuration="5.737273582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:28:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:28:44.412707155 +0000 UTC m=+829.033712368" watchObservedRunningTime="2026-11-02 01:28:47.737273582 +0000 UTC m=+832.358278806" Nov 02 01:28:47.800198 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:47.800165 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28" Nov 02 01:28:47.802703 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:47.801446 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28" Nov 02 01:28:47.803597 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:47.803573 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28" Nov 02 01:28:53.045219 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:53.045183 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" Nov 02 01:28:53.058546 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:53.058523 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" Nov 02 01:28:53.452427 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:53.452383 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" Nov 02 01:28:53.918070 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:53.918032 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb"] Nov 02 01:28:53.922477 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:53.922453 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb" Nov 02 01:28:53.935750 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:53.935726 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb"] Nov 02 01:28:54.027912 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.027876 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/917e0c7d-4eae-4d2d-bb59-f2de47fbfadb-workload-certs\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb\" (UID: \"917e0c7d-4eae-4d2d-bb59-f2de47fbfadb\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb" Nov 02 01:28:54.027912 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.027913 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/917e0c7d-4eae-4d2d-bb59-f2de47fbfadb-istio-token\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb\" (UID: \"917e0c7d-4eae-4d2d-bb59-f2de47fbfadb\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb" Nov 02 01:28:54.028163 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.027972 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/917e0c7d-4eae-4d2d-bb59-f2de47fbfadb-istiod-ca-cert\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb\" (UID: \"917e0c7d-4eae-4d2d-bb59-f2de47fbfadb\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb" Nov 02 01:28:54.028163 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.028110 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/917e0c7d-4eae-4d2d-bb59-f2de47fbfadb-credential-socket\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb\" (UID: \"917e0c7d-4eae-4d2d-bb59-f2de47fbfadb\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb" Nov 02 01:28:54.028163 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.028144 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzpc2\" (UniqueName: \"kubernetes.io/projected/917e0c7d-4eae-4d2d-bb59-f2de47fbfadb-kube-api-access-nzpc2\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb\" (UID: \"917e0c7d-4eae-4d2d-bb59-f2de47fbfadb\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb" Nov 02 01:28:54.028283 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.028264 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/917e0c7d-4eae-4d2d-bb59-f2de47fbfadb-istio-data\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb\" (UID: \"917e0c7d-4eae-4d2d-bb59-f2de47fbfadb\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb" Nov 02 01:28:54.028320 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.028298 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/917e0c7d-4eae-4d2d-bb59-f2de47fbfadb-workload-socket\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb\" (UID: \"917e0c7d-4eae-4d2d-bb59-f2de47fbfadb\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb" Nov 02 01:28:54.028363 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.028322 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/917e0c7d-4eae-4d2d-bb59-f2de47fbfadb-istio-envoy\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb\" (UID: \"917e0c7d-4eae-4d2d-bb59-f2de47fbfadb\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb" Nov 02 01:28:54.028454 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.028442 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/917e0c7d-4eae-4d2d-bb59-f2de47fbfadb-istio-podinfo\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb\" (UID: \"917e0c7d-4eae-4d2d-bb59-f2de47fbfadb\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb" Nov 02 01:28:54.130065 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.130032 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/917e0c7d-4eae-4d2d-bb59-f2de47fbfadb-istiod-ca-cert\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb\" (UID: \"917e0c7d-4eae-4d2d-bb59-f2de47fbfadb\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb" Nov 02 01:28:54.130065 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.130070 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/917e0c7d-4eae-4d2d-bb59-f2de47fbfadb-credential-socket\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb\" (UID: \"917e0c7d-4eae-4d2d-bb59-f2de47fbfadb\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb" Nov 02 01:28:54.130827 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.130087 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzpc2\" (UniqueName: \"kubernetes.io/projected/917e0c7d-4eae-4d2d-bb59-f2de47fbfadb-kube-api-access-nzpc2\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb\" (UID: \"917e0c7d-4eae-4d2d-bb59-f2de47fbfadb\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb" Nov 02 01:28:54.130827 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.130156 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/917e0c7d-4eae-4d2d-bb59-f2de47fbfadb-istio-data\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb\" (UID: \"917e0c7d-4eae-4d2d-bb59-f2de47fbfadb\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb" Nov 02 01:28:54.130827 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.130180 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/917e0c7d-4eae-4d2d-bb59-f2de47fbfadb-workload-socket\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb\" (UID: \"917e0c7d-4eae-4d2d-bb59-f2de47fbfadb\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb" Nov 02 01:28:54.130827 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.130195 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/917e0c7d-4eae-4d2d-bb59-f2de47fbfadb-istio-envoy\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb\" (UID: \"917e0c7d-4eae-4d2d-bb59-f2de47fbfadb\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb" Nov 02 01:28:54.130827 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.130260 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/917e0c7d-4eae-4d2d-bb59-f2de47fbfadb-istio-podinfo\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb\" (UID: \"917e0c7d-4eae-4d2d-bb59-f2de47fbfadb\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb" Nov 02 01:28:54.130827 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.130330 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/917e0c7d-4eae-4d2d-bb59-f2de47fbfadb-workload-certs\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb\" (UID: \"917e0c7d-4eae-4d2d-bb59-f2de47fbfadb\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb" Nov 02 01:28:54.130827 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.130362 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/917e0c7d-4eae-4d2d-bb59-f2de47fbfadb-istio-token\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb\" (UID: \"917e0c7d-4eae-4d2d-bb59-f2de47fbfadb\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb" Nov 02 01:28:54.130827 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.130732 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/917e0c7d-4eae-4d2d-bb59-f2de47fbfadb-workload-certs\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb\" (UID: \"917e0c7d-4eae-4d2d-bb59-f2de47fbfadb\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb" Nov 02 01:28:54.130827 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.130731 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/917e0c7d-4eae-4d2d-bb59-f2de47fbfadb-workload-socket\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb\" (UID: \"917e0c7d-4eae-4d2d-bb59-f2de47fbfadb\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb" Nov 02 01:28:54.130827 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.130798 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/917e0c7d-4eae-4d2d-bb59-f2de47fbfadb-credential-socket\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb\" (UID: \"917e0c7d-4eae-4d2d-bb59-f2de47fbfadb\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb" Nov 02 01:28:54.131310 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.131154 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/917e0c7d-4eae-4d2d-bb59-f2de47fbfadb-istiod-ca-cert\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb\" (UID: \"917e0c7d-4eae-4d2d-bb59-f2de47fbfadb\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb" Nov 02 01:28:54.131346 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.131312 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/917e0c7d-4eae-4d2d-bb59-f2de47fbfadb-istio-data\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb\" (UID: \"917e0c7d-4eae-4d2d-bb59-f2de47fbfadb\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb" Nov 02 01:28:54.133215 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.133191 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/917e0c7d-4eae-4d2d-bb59-f2de47fbfadb-istio-podinfo\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb\" (UID: \"917e0c7d-4eae-4d2d-bb59-f2de47fbfadb\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb" Nov 02 01:28:54.133359 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.133339 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/917e0c7d-4eae-4d2d-bb59-f2de47fbfadb-istio-envoy\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb\" (UID: \"917e0c7d-4eae-4d2d-bb59-f2de47fbfadb\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb" Nov 02 01:28:54.144555 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.144529 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/917e0c7d-4eae-4d2d-bb59-f2de47fbfadb-istio-token\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb\" (UID: \"917e0c7d-4eae-4d2d-bb59-f2de47fbfadb\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb" Nov 02 01:28:54.144692 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.144595 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzpc2\" (UniqueName: \"kubernetes.io/projected/917e0c7d-4eae-4d2d-bb59-f2de47fbfadb-kube-api-access-nzpc2\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb\" (UID: \"917e0c7d-4eae-4d2d-bb59-f2de47fbfadb\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb" Nov 02 01:28:54.149200 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.149177 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s"] Nov 02 01:28:54.155083 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.155059 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s" Nov 02 01:28:54.169589 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.169528 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s"] Nov 02 01:28:54.232010 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.231975 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a7a8380-cdbe-4204-a374-8d42f30b87d7-istio-podinfo\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s\" (UID: \"9a7a8380-cdbe-4204-a374-8d42f30b87d7\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s" Nov 02 01:28:54.232185 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.232047 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a7a8380-cdbe-4204-a374-8d42f30b87d7-workload-certs\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s\" (UID: \"9a7a8380-cdbe-4204-a374-8d42f30b87d7\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s" Nov 02 01:28:54.232185 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.232160 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a7a8380-cdbe-4204-a374-8d42f30b87d7-istio-envoy\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s\" (UID: \"9a7a8380-cdbe-4204-a374-8d42f30b87d7\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s" Nov 02 01:28:54.232306 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.232257 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7a8380-cdbe-4204-a374-8d42f30b87d7-workload-socket\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s\" (UID: \"9a7a8380-cdbe-4204-a374-8d42f30b87d7\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s" Nov 02 01:28:54.232306 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.232296 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkqnd\" (UniqueName: \"kubernetes.io/projected/9a7a8380-cdbe-4204-a374-8d42f30b87d7-kube-api-access-gkqnd\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s\" (UID: \"9a7a8380-cdbe-4204-a374-8d42f30b87d7\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s" Nov 02 01:28:54.232421 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.232357 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a7a8380-cdbe-4204-a374-8d42f30b87d7-istio-data\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s\" (UID: \"9a7a8380-cdbe-4204-a374-8d42f30b87d7\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s" Nov 02 01:28:54.232421 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.232410 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a7a8380-cdbe-4204-a374-8d42f30b87d7-istiod-ca-cert\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s\" (UID: \"9a7a8380-cdbe-4204-a374-8d42f30b87d7\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s" Nov 02 01:28:54.232540 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.232520 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a7a8380-cdbe-4204-a374-8d42f30b87d7-istio-token\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s\" (UID: \"9a7a8380-cdbe-4204-a374-8d42f30b87d7\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s" Nov 02 01:28:54.232609 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.232591 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7a8380-cdbe-4204-a374-8d42f30b87d7-credential-socket\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s\" (UID: \"9a7a8380-cdbe-4204-a374-8d42f30b87d7\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s" Nov 02 01:28:54.238482 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.238458 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb" Nov 02 01:28:54.334262 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.334227 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a7a8380-cdbe-4204-a374-8d42f30b87d7-istio-token\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s\" (UID: \"9a7a8380-cdbe-4204-a374-8d42f30b87d7\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s" Nov 02 01:28:54.334434 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.334302 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7a8380-cdbe-4204-a374-8d42f30b87d7-credential-socket\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s\" (UID: \"9a7a8380-cdbe-4204-a374-8d42f30b87d7\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s" Nov 02 01:28:54.334474 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.334460 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a7a8380-cdbe-4204-a374-8d42f30b87d7-istio-podinfo\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s\" (UID: \"9a7a8380-cdbe-4204-a374-8d42f30b87d7\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s" Nov 02 01:28:54.334532 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.334516 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a7a8380-cdbe-4204-a374-8d42f30b87d7-workload-certs\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s\" (UID: \"9a7a8380-cdbe-4204-a374-8d42f30b87d7\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s" Nov 02 01:28:54.334569 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.334558 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a7a8380-cdbe-4204-a374-8d42f30b87d7-istio-envoy\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s\" (UID: \"9a7a8380-cdbe-4204-a374-8d42f30b87d7\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s" Nov 02 01:28:54.334635 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.334621 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7a8380-cdbe-4204-a374-8d42f30b87d7-workload-socket\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s\" (UID: \"9a7a8380-cdbe-4204-a374-8d42f30b87d7\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s" Nov 02 01:28:54.334686 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.334672 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkqnd\" (UniqueName: \"kubernetes.io/projected/9a7a8380-cdbe-4204-a374-8d42f30b87d7-kube-api-access-gkqnd\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s\" (UID: \"9a7a8380-cdbe-4204-a374-8d42f30b87d7\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s" Nov 02 01:28:54.334740 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.334727 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a7a8380-cdbe-4204-a374-8d42f30b87d7-istio-data\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s\" (UID: \"9a7a8380-cdbe-4204-a374-8d42f30b87d7\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s" Nov 02 01:28:54.334775 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.334764 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a7a8380-cdbe-4204-a374-8d42f30b87d7-istiod-ca-cert\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s\" (UID: \"9a7a8380-cdbe-4204-a374-8d42f30b87d7\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s" Nov 02 01:28:54.334865 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.334838 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7a8380-cdbe-4204-a374-8d42f30b87d7-credential-socket\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s\" (UID: \"9a7a8380-cdbe-4204-a374-8d42f30b87d7\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s" Nov 02 01:28:54.334928 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.334910 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a7a8380-cdbe-4204-a374-8d42f30b87d7-workload-certs\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s\" (UID: \"9a7a8380-cdbe-4204-a374-8d42f30b87d7\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s" Nov 02 01:28:54.335188 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.335168 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7a8380-cdbe-4204-a374-8d42f30b87d7-workload-socket\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s\" (UID: \"9a7a8380-cdbe-4204-a374-8d42f30b87d7\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s" Nov 02 01:28:54.335613 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.335589 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a7a8380-cdbe-4204-a374-8d42f30b87d7-istiod-ca-cert\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s\" (UID: \"9a7a8380-cdbe-4204-a374-8d42f30b87d7\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s" Nov 02 01:28:54.335750 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.335727 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a7a8380-cdbe-4204-a374-8d42f30b87d7-istio-data\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s\" (UID: \"9a7a8380-cdbe-4204-a374-8d42f30b87d7\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s" Nov 02 01:28:54.337628 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.337597 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a7a8380-cdbe-4204-a374-8d42f30b87d7-istio-envoy\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s\" (UID: \"9a7a8380-cdbe-4204-a374-8d42f30b87d7\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s" Nov 02 01:28:54.337802 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.337782 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a7a8380-cdbe-4204-a374-8d42f30b87d7-istio-podinfo\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s\" (UID: \"9a7a8380-cdbe-4204-a374-8d42f30b87d7\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s" Nov 02 01:28:54.349411 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.348472 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkqnd\" (UniqueName: \"kubernetes.io/projected/9a7a8380-cdbe-4204-a374-8d42f30b87d7-kube-api-access-gkqnd\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s\" (UID: \"9a7a8380-cdbe-4204-a374-8d42f30b87d7\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s" Nov 02 01:28:54.349411 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.348814 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a7a8380-cdbe-4204-a374-8d42f30b87d7-istio-token\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s\" (UID: \"9a7a8380-cdbe-4204-a374-8d42f30b87d7\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s" Nov 02 01:28:54.445007 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.444891 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c"] Nov 02 01:28:54.451570 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.451543 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c" Nov 02 01:28:54.464445 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.464415 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c"] Nov 02 01:28:54.475690 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.475619 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s" Nov 02 01:28:54.477294 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.477268 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4"] Nov 02 01:28:54.486941 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.486915 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4" Nov 02 01:28:54.495563 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.495538 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4"] Nov 02 01:28:54.536893 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.536162 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb"] Nov 02 01:28:54.536893 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.536784 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9-istiod-ca-cert\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4\" (UID: \"9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4" Nov 02 01:28:54.536893 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.536828 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pssnc\" (UniqueName: \"kubernetes.io/projected/9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9-kube-api-access-pssnc\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4\" (UID: \"9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4" Nov 02 01:28:54.537072 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.536914 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c37ba466-a830-4246-832b-a8a4d0b5ab9f-istio-podinfo\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c\" (UID: \"c37ba466-a830-4246-832b-a8a4d0b5ab9f\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c" Nov 02 01:28:54.537072 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.537013 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c37ba466-a830-4246-832b-a8a4d0b5ab9f-istiod-ca-cert\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c\" (UID: \"c37ba466-a830-4246-832b-a8a4d0b5ab9f\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c" Nov 02 01:28:54.537072 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.537055 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c37ba466-a830-4246-832b-a8a4d0b5ab9f-istio-envoy\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c\" (UID: \"c37ba466-a830-4246-832b-a8a4d0b5ab9f\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c" Nov 02 01:28:54.537221 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.537099 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx9f9\" (UniqueName: \"kubernetes.io/projected/c37ba466-a830-4246-832b-a8a4d0b5ab9f-kube-api-access-gx9f9\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c\" (UID: \"c37ba466-a830-4246-832b-a8a4d0b5ab9f\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c" Nov 02 01:28:54.537440 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.537282 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9-credential-socket\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4\" (UID: \"9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4" Nov 02 01:28:54.537440 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.537323 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c37ba466-a830-4246-832b-a8a4d0b5ab9f-istio-token\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c\" (UID: \"c37ba466-a830-4246-832b-a8a4d0b5ab9f\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c" Nov 02 01:28:54.537440 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.537402 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9-istio-data\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4\" (UID: \"9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4" Nov 02 01:28:54.537440 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.537422 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9-workload-socket\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4\" (UID: \"9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4" Nov 02 01:28:54.537715 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.537695 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9-workload-certs\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4\" (UID: \"9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4" Nov 02 01:28:54.537750 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.537735 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9-istio-podinfo\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4\" (UID: \"9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4" Nov 02 01:28:54.537787 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.537762 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9-istio-envoy\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4\" (UID: \"9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4" Nov 02 01:28:54.537821 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.537787 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c37ba466-a830-4246-832b-a8a4d0b5ab9f-credential-socket\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c\" (UID: \"c37ba466-a830-4246-832b-a8a4d0b5ab9f\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c" Nov 02 01:28:54.537857 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.537832 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c37ba466-a830-4246-832b-a8a4d0b5ab9f-workload-socket\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c\" (UID: \"c37ba466-a830-4246-832b-a8a4d0b5ab9f\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c" Nov 02 01:28:54.537857 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.537848 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c37ba466-a830-4246-832b-a8a4d0b5ab9f-workload-certs\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c\" (UID: \"c37ba466-a830-4246-832b-a8a4d0b5ab9f\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c" Nov 02 01:28:54.537917 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.537901 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c37ba466-a830-4246-832b-a8a4d0b5ab9f-istio-data\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c\" (UID: \"c37ba466-a830-4246-832b-a8a4d0b5ab9f\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c" Nov 02 01:28:54.537949 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.537922 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9-istio-token\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4\" (UID: \"9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4" Nov 02 01:28:54.539545 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:28:54.539516 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod917e0c7d_4eae_4d2d_bb59_f2de47fbfadb.slice/crio-6d58a54b89213f35dbac1aeead851b72347cf109337710d125e6f4b7056af954 WatchSource:0}: Error finding container 6d58a54b89213f35dbac1aeead851b72347cf109337710d125e6f4b7056af954: Status 404 returned error can't find the container with id 6d58a54b89213f35dbac1aeead851b72347cf109337710d125e6f4b7056af954 Nov 02 01:28:54.542136 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.542102 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:28:54.542240 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.542162 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:28:54.542240 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.542188 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:28:54.646226 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.640414 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9-istio-data\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4\" (UID: \"9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4" Nov 02 01:28:54.646226 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.640509 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9-istio-data\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4\" (UID: \"9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4" Nov 02 01:28:54.646226 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.640585 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9-workload-socket\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4\" (UID: \"9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4" Nov 02 01:28:54.646226 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.640647 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9-workload-certs\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4\" (UID: \"9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4" Nov 02 01:28:54.646226 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.641114 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9-workload-socket\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4\" (UID: \"9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4" Nov 02 01:28:54.646226 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.641839 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9-workload-certs\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4\" (UID: \"9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4" Nov 02 01:28:54.646226 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.641980 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9-istio-podinfo\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4\" (UID: \"9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4" Nov 02 01:28:54.646226 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.642103 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9-istio-envoy\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4\" (UID: \"9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4" Nov 02 01:28:54.646226 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.642701 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c37ba466-a830-4246-832b-a8a4d0b5ab9f-credential-socket\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c\" (UID: \"c37ba466-a830-4246-832b-a8a4d0b5ab9f\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c" Nov 02 01:28:54.646226 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.642771 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c37ba466-a830-4246-832b-a8a4d0b5ab9f-workload-socket\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c\" (UID: \"c37ba466-a830-4246-832b-a8a4d0b5ab9f\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c" Nov 02 01:28:54.646226 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.642799 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c37ba466-a830-4246-832b-a8a4d0b5ab9f-workload-certs\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c\" (UID: \"c37ba466-a830-4246-832b-a8a4d0b5ab9f\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c" Nov 02 01:28:54.646226 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.642869 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c37ba466-a830-4246-832b-a8a4d0b5ab9f-istio-data\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c\" (UID: \"c37ba466-a830-4246-832b-a8a4d0b5ab9f\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c" Nov 02 01:28:54.646226 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.642900 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9-istio-token\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4\" (UID: \"9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4" Nov 02 01:28:54.646226 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.642974 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9-istiod-ca-cert\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4\" (UID: \"9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4" Nov 02 01:28:54.646226 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.643050 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pssnc\" (UniqueName: \"kubernetes.io/projected/9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9-kube-api-access-pssnc\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4\" (UID: \"9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4" Nov 02 01:28:54.646226 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.643099 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c37ba466-a830-4246-832b-a8a4d0b5ab9f-istio-podinfo\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c\" (UID: \"c37ba466-a830-4246-832b-a8a4d0b5ab9f\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c" Nov 02 01:28:54.646226 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.643138 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c37ba466-a830-4246-832b-a8a4d0b5ab9f-istiod-ca-cert\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c\" (UID: \"c37ba466-a830-4246-832b-a8a4d0b5ab9f\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c" Nov 02 01:28:54.646226 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.643167 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c37ba466-a830-4246-832b-a8a4d0b5ab9f-istio-envoy\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c\" (UID: \"c37ba466-a830-4246-832b-a8a4d0b5ab9f\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c" Nov 02 01:28:54.646226 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.643722 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c37ba466-a830-4246-832b-a8a4d0b5ab9f-workload-certs\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c\" (UID: \"c37ba466-a830-4246-832b-a8a4d0b5ab9f\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c" Nov 02 01:28:54.646226 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.643999 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c37ba466-a830-4246-832b-a8a4d0b5ab9f-workload-socket\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c\" (UID: \"c37ba466-a830-4246-832b-a8a4d0b5ab9f\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c" Nov 02 01:28:54.646226 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.644910 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c37ba466-a830-4246-832b-a8a4d0b5ab9f-credential-socket\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c\" (UID: \"c37ba466-a830-4246-832b-a8a4d0b5ab9f\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c" Nov 02 01:28:54.646226 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.645490 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9-istiod-ca-cert\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4\" (UID: \"9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4" Nov 02 01:28:54.646226 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.645730 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c37ba466-a830-4246-832b-a8a4d0b5ab9f-istio-data\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c\" (UID: \"c37ba466-a830-4246-832b-a8a4d0b5ab9f\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c" Nov 02 01:28:54.647682 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.646330 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx9f9\" (UniqueName: \"kubernetes.io/projected/c37ba466-a830-4246-832b-a8a4d0b5ab9f-kube-api-access-gx9f9\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c\" (UID: \"c37ba466-a830-4246-832b-a8a4d0b5ab9f\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c" Nov 02 01:28:54.647682 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.646398 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9-credential-socket\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4\" (UID: \"9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4" Nov 02 01:28:54.647682 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.646430 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c37ba466-a830-4246-832b-a8a4d0b5ab9f-istio-token\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c\" (UID: \"c37ba466-a830-4246-832b-a8a4d0b5ab9f\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c" Nov 02 01:28:54.647682 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.647223 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9-credential-socket\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4\" (UID: \"9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4" Nov 02 01:28:54.650150 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.648673 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c37ba466-a830-4246-832b-a8a4d0b5ab9f-istio-envoy\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c\" (UID: \"c37ba466-a830-4246-832b-a8a4d0b5ab9f\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c" Nov 02 01:28:54.650150 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.648747 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c37ba466-a830-4246-832b-a8a4d0b5ab9f-istiod-ca-cert\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c\" (UID: \"c37ba466-a830-4246-832b-a8a4d0b5ab9f\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c" Nov 02 01:28:54.656026 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.655984 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c37ba466-a830-4246-832b-a8a4d0b5ab9f-istio-podinfo\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c\" (UID: \"c37ba466-a830-4246-832b-a8a4d0b5ab9f\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c" Nov 02 01:28:54.659409 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.658950 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9-istio-podinfo\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4\" (UID: \"9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4" Nov 02 01:28:54.659710 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.659686 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9-istio-envoy\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4\" (UID: \"9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4" Nov 02 01:28:54.677676 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.663111 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pssnc\" (UniqueName: \"kubernetes.io/projected/9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9-kube-api-access-pssnc\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4\" (UID: \"9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4" Nov 02 01:28:54.680624 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.680573 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx9f9\" (UniqueName: \"kubernetes.io/projected/c37ba466-a830-4246-832b-a8a4d0b5ab9f-kube-api-access-gx9f9\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c\" (UID: \"c37ba466-a830-4246-832b-a8a4d0b5ab9f\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c" Nov 02 01:28:54.682441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.682218 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c37ba466-a830-4246-832b-a8a4d0b5ab9f-istio-token\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c\" (UID: \"c37ba466-a830-4246-832b-a8a4d0b5ab9f\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c" Nov 02 01:28:54.683425 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.683399 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9-istio-token\") pod \"e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4\" (UID: \"9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9\") " pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4" Nov 02 01:28:54.782590 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.782512 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c" Nov 02 01:28:54.813870 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.813841 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4" Nov 02 01:28:54.846730 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.846688 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s"] Nov 02 01:28:54.854927 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.854338 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:28:54.854927 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.854439 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:28:54.854927 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:54.854486 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:28:55.089875 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:55.089847 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c"] Nov 02 01:28:55.092128 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:28:55.092102 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc37ba466_a830_4246_832b_a8a4d0b5ab9f.slice/crio-8adb3a232408bf710b9bcef8bf770878b59e42ee9287210113b732ddd99ae1c2 WatchSource:0}: Error finding container 8adb3a232408bf710b9bcef8bf770878b59e42ee9287210113b732ddd99ae1c2: Status 404 returned error can't find the container with id 8adb3a232408bf710b9bcef8bf770878b59e42ee9287210113b732ddd99ae1c2 Nov 02 01:28:55.095191 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:55.095158 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:28:55.095282 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:55.095221 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:28:55.095282 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:55.095246 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:28:55.140271 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:55.140244 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4"] Nov 02 01:28:55.142667 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:28:55.142639 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b13ff7b_15dc_4a12_882d_6c0fc8fe25d9.slice/crio-dcd8992480701c66624cab73d59ab311737b7094b8bb56a8292e97aabfd7f5f8 WatchSource:0}: Error finding container dcd8992480701c66624cab73d59ab311737b7094b8bb56a8292e97aabfd7f5f8: Status 404 returned error can't find the container with id dcd8992480701c66624cab73d59ab311737b7094b8bb56a8292e97aabfd7f5f8 Nov 02 01:28:55.145742 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:55.145705 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:28:55.145831 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:55.145783 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:28:55.145831 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:55.145813 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:28:55.469992 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:55.469942 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb" event={"ID":"917e0c7d-4eae-4d2d-bb59-f2de47fbfadb","Type":"ContainerStarted","Data":"ae0e647af21cce922369cf85ae035291361db5082678af5fe817f4f890c8389b"} Nov 02 01:28:55.470170 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:55.470041 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb" event={"ID":"917e0c7d-4eae-4d2d-bb59-f2de47fbfadb","Type":"ContainerStarted","Data":"6d58a54b89213f35dbac1aeead851b72347cf109337710d125e6f4b7056af954"} Nov 02 01:28:55.472445 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:55.472417 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4" event={"ID":"9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9","Type":"ContainerStarted","Data":"3568ce28623faffcd65b62ab78a362f36ed1388d24cb534191d8cc7ce3f7c87b"} Nov 02 01:28:55.472581 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:55.472469 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4" event={"ID":"9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9","Type":"ContainerStarted","Data":"dcd8992480701c66624cab73d59ab311737b7094b8bb56a8292e97aabfd7f5f8"} Nov 02 01:28:55.475866 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:55.475837 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c" event={"ID":"c37ba466-a830-4246-832b-a8a4d0b5ab9f","Type":"ContainerStarted","Data":"73d46afa0f9cf0460c0eb36b5ffc4af31c63614272d021f283b01568394d56f8"} Nov 02 01:28:55.476011 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:55.475912 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c" event={"ID":"c37ba466-a830-4246-832b-a8a4d0b5ab9f","Type":"ContainerStarted","Data":"8adb3a232408bf710b9bcef8bf770878b59e42ee9287210113b732ddd99ae1c2"} Nov 02 01:28:55.478303 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:55.478280 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s" event={"ID":"9a7a8380-cdbe-4204-a374-8d42f30b87d7","Type":"ContainerStarted","Data":"077360db8c1c94db2557c7917d6924ddd76d531b22434a58f7584e40ad8c79c7"} Nov 02 01:28:55.478499 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:55.478319 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s" event={"ID":"9a7a8380-cdbe-4204-a374-8d42f30b87d7","Type":"ContainerStarted","Data":"40570b5065a75d0fc0429f07f80acceb1f98021703409e439fe451fa5b1c8955"} Nov 02 01:28:55.494873 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:55.494806 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb" podStartSLOduration=2.494787218 podStartE2EDuration="2.494787218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:28:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:28:55.49415733 +0000 UTC m=+840.115162540" watchObservedRunningTime="2026-11-02 01:28:55.494787218 +0000 UTC m=+840.115792430" Nov 02 01:28:55.566858 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:55.566800 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s" podStartSLOduration=1.5667833340000001 podStartE2EDuration="1.566783334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:28:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:28:55.537086128 +0000 UTC m=+840.158091339" watchObservedRunningTime="2026-11-02 01:28:55.566783334 +0000 UTC m=+840.187788546" Nov 02 01:28:55.619011 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:55.618935 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4" podStartSLOduration=1.618914856 podStartE2EDuration="1.618914856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:28:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:28:55.605757923 +0000 UTC m=+840.226763136" watchObservedRunningTime="2026-11-02 01:28:55.618914856 +0000 UTC m=+840.239920063" Nov 02 01:28:55.624755 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:55.624717 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c" podStartSLOduration=1.624706924 podStartE2EDuration="1.624706924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:28:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:28:55.583836985 +0000 UTC m=+840.204842196" watchObservedRunningTime="2026-11-02 01:28:55.624706924 +0000 UTC m=+840.245712135" Nov 02 01:28:55.789739 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:55.789618 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c" Nov 02 01:28:55.814584 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:55.814551 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4" Nov 02 01:28:56.256402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:56.249437 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb" Nov 02 01:28:56.256402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:56.255497 4427 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 02 01:28:56.256402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:56.255560 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb" podUID="917e0c7d-4eae-4d2d-bb59-f2de47fbfadb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 02 01:28:56.485412 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:56.483767 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s" Nov 02 01:28:56.516489 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:56.516366 4427 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 02 01:28:56.516964 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:56.516932 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s" podUID="9a7a8380-cdbe-4204-a374-8d42f30b87d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 02 01:28:56.791839 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:56.791748 4427 patch_prober.go:29] interesting pod/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 02 01:28:56.792073 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:56.792047 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c" podUID="c37ba466-a830-4246-832b-a8a4d0b5ab9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 02 01:28:56.822440 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:56.821112 4427 patch_prober.go:29] interesting pod/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 02 01:28:56.822440 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:56.821171 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4" podUID="9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 02 01:28:57.226399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:57.226314 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" podUID="cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:28:57.269399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:57.259963 4427 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 02 01:28:57.269399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:57.260023 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb" podUID="917e0c7d-4eae-4d2d-bb59-f2de47fbfadb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 02 01:28:57.436401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:57.433482 4427 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" podUID="cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 01:28:57.447400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:57.446746 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" podUID="47a9069f-1ffa-4c83-be8b-5e62f1d3275a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:28:57.480287 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:57.479777 4427 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 02 01:28:57.480287 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:57.479840 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s" podUID="9a7a8380-cdbe-4204-a374-8d42f30b87d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 02 01:28:57.560754 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:57.560704 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" podUID="d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Nov 02 01:28:57.622483 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:57.615631 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" podUID="eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:28:57.675439 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:57.663688 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" podUID="d73ae484-a9b8-4565-a346-0d803a1981ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:28:57.675439 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:57.664516 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" podUID="fb533b5f-3525-454a-8ce2-ea9246194f28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Nov 02 01:28:57.694402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:57.688319 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" podUID="7085c0ab-7ee3-4665-abad-2977c25f68ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:28:57.694402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:57.690673 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" podUID="ea61971c-5a72-4609-b5a0-58a74b324bb7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:28:57.718544 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:57.710854 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" podUID="c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:28:57.763487 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:57.760211 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" podUID="9c056798-2cf8-47b1-9829-67cebda7a248" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:28:57.803994 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:57.803948 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" podUID="706df1bd-19cf-4ab9-9aaa-3c12c97ccf45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:28:57.811932 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:57.811006 4427 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" podUID="9c056798-2cf8-47b1-9829-67cebda7a248" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": context deadline exceeded" Nov 02 01:28:57.812162 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:57.811743 4427 patch_prober.go:29] interesting pod/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 02 01:28:57.816398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:57.812327 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c" podUID="c37ba466-a830-4246-832b-a8a4d0b5ab9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 02 01:28:57.903401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:57.888048 4427 patch_prober.go:29] interesting pod/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 02 01:28:57.903401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:57.888122 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4" podUID="9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 02 01:28:57.903401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:57.899457 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" podUID="353a8a41-21ee-4470-861c-b53833beb106" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:28:58.019458 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:58.018241 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" podUID="f67c954a-ce5b-41d9-8d99-7d126bd89d59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:28:58.088087 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:58.087462 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" podUID="366c78ee-4aaf-4b7c-84b4-17028538cb1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:28:58.106400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:58.095111 4427 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" podUID="fb533b5f-3525-454a-8ce2-ea9246194f28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:28:58.228499 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:58.228451 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" podUID="cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 02 01:28:58.229400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:58.229357 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" Nov 02 01:28:58.238920 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:58.238890 4427 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 02 01:28:58.239233 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:58.239205 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb" podUID="917e0c7d-4eae-4d2d-bb59-f2de47fbfadb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 02 01:28:58.859533 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:58.859484 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" podUID="9c056798-2cf8-47b1-9829-67cebda7a248" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:28:58.905974 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:58.905941 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" Nov 02 01:28:58.969401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:58.949602 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" podUID="d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Nov 02 01:28:58.969768 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:58.969731 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" podUID="706df1bd-19cf-4ab9-9aaa-3c12c97ccf45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 02 01:28:58.970205 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:58.957624 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" podUID="eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:28:58.970646 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:58.970627 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" Nov 02 01:28:58.970829 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:58.970815 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" Nov 02 01:28:58.970951 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:58.970941 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" Nov 02 01:28:58.990411 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:58.971914 4427 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" podUID="c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": context deadline exceeded" Nov 02 01:28:58.990652 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:58.977051 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" podUID="d73ae484-a9b8-4565-a346-0d803a1981ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": context deadline exceeded" Nov 02 01:28:59.000863 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:58.977110 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" podUID="7085c0ab-7ee3-4665-abad-2977c25f68ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Nov 02 01:28:59.026400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:59.001355 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" Nov 02 01:28:59.026711 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:59.026687 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" Nov 02 01:28:59.031399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:58.978723 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" podUID="353a8a41-21ee-4470-861c-b53833beb106" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:28:59.069400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:58.977132 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" podUID="fb533b5f-3525-454a-8ce2-ea9246194f28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Nov 02 01:28:59.069400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:59.057045 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" Nov 02 01:28:59.069400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:59.057103 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" Nov 02 01:28:59.069400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:58.979591 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" podUID="c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Nov 02 01:28:59.069400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:59.033190 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" podUID="f67c954a-ce5b-41d9-8d99-7d126bd89d59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:28:59.069400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:59.059147 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" Nov 02 01:28:59.069400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:59.059193 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" Nov 02 01:28:59.133500 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:59.133410 4427 patch_prober.go:29] interesting pod/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 02 01:28:59.258403 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:59.233442 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4" podUID="9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 02 01:28:59.268401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:59.230001 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" podUID="cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:28:59.268401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:59.171547 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" podUID="366c78ee-4aaf-4b7c-84b4-17028538cb1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:28:59.268401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:59.136786 4427 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 02 01:28:59.268401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:59.266025 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s" podUID="9a7a8380-cdbe-4204-a374-8d42f30b87d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 02 01:28:59.268401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:59.266117 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" Nov 02 01:28:59.268401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:59.138663 4427 patch_prober.go:29] interesting pod/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 02 01:28:59.268401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:59.267008 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c" podUID="c37ba466-a830-4246-832b-a8a4d0b5ab9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 02 01:28:59.268401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:59.258764 4427 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 02 01:28:59.268401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:59.267058 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb" podUID="917e0c7d-4eae-4d2d-bb59-f2de47fbfadb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 02 01:28:59.494450 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:59.476794 4427 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 02 01:28:59.494450 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:59.476856 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s" podUID="9a7a8380-cdbe-4204-a374-8d42f30b87d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 02 01:28:59.778423 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:59.778311 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" podUID="47a9069f-1ffa-4c83-be8b-5e62f1d3275a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:28:59.791425 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:59.786089 4427 patch_prober.go:29] interesting pod/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 02 01:28:59.791425 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:59.786142 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c" podUID="c37ba466-a830-4246-832b-a8a4d0b5ab9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 02 01:28:59.815808 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:59.815773 4427 patch_prober.go:29] interesting pod/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 02 01:28:59.819401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:59.816039 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4" podUID="9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 02 01:28:59.952083 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:59.933029 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" podUID="9c056798-2cf8-47b1-9829-67cebda7a248" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:28:59.987485 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:59.983928 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" podUID="eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Nov 02 01:28:59.996537 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:59.989554 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" podUID="d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Nov 02 01:28:59.996537 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:28:59.993628 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" podUID="706df1bd-19cf-4ab9-9aaa-3c12c97ccf45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 02 01:29:00.016490 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:00.012471 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" podUID="7085c0ab-7ee3-4665-abad-2977c25f68ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Nov 02 01:29:00.034564 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:00.025939 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" podUID="d73ae484-a9b8-4565-a346-0d803a1981ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": context deadline exceeded" Nov 02 01:29:00.041872 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:00.041830 4427 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-nddkv" podUID="4379e4b2-e68b-4ddf-a3bf-9ec404e61f56" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.18:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:00.111626 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:00.090173 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" podUID="f67c954a-ce5b-41d9-8d99-7d126bd89d59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:00.118967 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:00.118919 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" podUID="353a8a41-21ee-4470-861c-b53833beb106" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:00.145497 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:00.145434 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" podUID="fb533b5f-3525-454a-8ce2-ea9246194f28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Nov 02 01:29:00.151691 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:00.145997 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" podUID="c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Nov 02 01:29:00.273413 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:00.271171 4427 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 02 01:29:00.273413 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:00.271232 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb" podUID="917e0c7d-4eae-4d2d-bb59-f2de47fbfadb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 02 01:29:00.273413 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:00.272330 4427 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" podUID="eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:00.273413 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:00.272450 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" podUID="366c78ee-4aaf-4b7c-84b4-17028538cb1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:00.273817 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:00.273779 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" podUID="cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 02 01:29:00.504443 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:00.504406 4427 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 02 01:29:00.537501 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:00.537456 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s" podUID="9a7a8380-cdbe-4204-a374-8d42f30b87d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 02 01:29:00.656683 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:00.656584 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" podUID="ea61971c-5a72-4609-b5a0-58a74b324bb7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 02 01:29:00.801402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:00.781283 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" podUID="47a9069f-1ffa-4c83-be8b-5e62f1d3275a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:00.801402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:00.782125 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" Nov 02 01:29:00.829537 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:00.828655 4427 patch_prober.go:29] interesting pod/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 02 01:29:00.829537 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:00.828758 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4" podUID="9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 02 01:29:00.866403 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:00.851835 4427 patch_prober.go:29] interesting pod/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 02 01:29:00.866403 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:00.851916 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c" podUID="c37ba466-a830-4246-832b-a8a4d0b5ab9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 02 01:29:00.871118 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:00.871089 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" Nov 02 01:29:00.955534 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:00.955440 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" podUID="9c056798-2cf8-47b1-9829-67cebda7a248" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:00.982744 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:00.982713 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" Nov 02 01:29:01.077490 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:00.990465 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" podUID="d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Nov 02 01:29:01.097417 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:00.994445 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" podUID="706df1bd-19cf-4ab9-9aaa-3c12c97ccf45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 02 01:29:01.120953 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:01.026500 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" podUID="d73ae484-a9b8-4565-a346-0d803a1981ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:01.297457 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:01.026534 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" podUID="7085c0ab-7ee3-4665-abad-2977c25f68ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:01.297730 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:01.092044 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" podUID="f67c954a-ce5b-41d9-8d99-7d126bd89d59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:01.298155 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:01.120750 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" podUID="353a8a41-21ee-4470-861c-b53833beb106" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 02 01:29:01.299018 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:01.154930 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" podUID="c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:01.300891 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:01.200692 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" podUID="fb533b5f-3525-454a-8ce2-ea9246194f28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:01.329404 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:01.249971 4427 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 02 01:29:01.348504 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:01.348459 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb" podUID="917e0c7d-4eae-4d2d-bb59-f2de47fbfadb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 02 01:29:01.348734 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:01.292517 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" podUID="366c78ee-4aaf-4b7c-84b4-17028538cb1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:01.481467 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:01.480914 4427 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 02 01:29:01.481467 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:01.480976 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s" podUID="9a7a8380-cdbe-4204-a374-8d42f30b87d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 02 01:29:01.789826 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:01.788461 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" podUID="47a9069f-1ffa-4c83-be8b-5e62f1d3275a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:01.789826 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:01.789514 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-nddkv" podUID="4379e4b2-e68b-4ddf-a3bf-9ec404e61f56" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.18:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:01.790061 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:01.789831 4427 patch_prober.go:29] interesting pod/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 02 01:29:01.790061 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:01.789866 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c" podUID="c37ba466-a830-4246-832b-a8a4d0b5ab9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 02 01:29:01.820420 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:01.819639 4427 patch_prober.go:29] interesting pod/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 02 01:29:01.820420 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:01.819700 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4" podUID="9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 02 01:29:01.820420 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:29:01.820331 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:29:01.970418 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:01.961583 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" podUID="9c056798-2cf8-47b1-9829-67cebda7a248" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:01.970418 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:01.962531 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" Nov 02 01:29:02.147076 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:02.136775 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" podUID="706df1bd-19cf-4ab9-9aaa-3c12c97ccf45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:02.256661 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:02.256612 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" podUID="d73ae484-a9b8-4565-a346-0d803a1981ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": context deadline exceeded" Nov 02 01:29:02.319788 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:02.269497 4427 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 02 01:29:02.320473 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:02.320441 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb" podUID="917e0c7d-4eae-4d2d-bb59-f2de47fbfadb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 02 01:29:02.328176 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:02.313730 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" podUID="c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:02.332795 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:02.313753 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" podUID="353a8a41-21ee-4470-861c-b53833beb106" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:02.345395 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:02.313788 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" podUID="fb533b5f-3525-454a-8ce2-ea9246194f28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:02.364411 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:02.363894 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" Nov 02 01:29:02.373410 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:02.364999 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" podUID="366c78ee-4aaf-4b7c-84b4-17028538cb1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:02.382529 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:02.365107 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" podUID="7085c0ab-7ee3-4665-abad-2977c25f68ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:02.404457 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:02.313647 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" podUID="f67c954a-ce5b-41d9-8d99-7d126bd89d59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:02.467520 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:02.452899 4427 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" podUID="d73ae484-a9b8-4565-a346-0d803a1981ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:02.668475 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:02.506840 4427 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 02 01:29:02.668475 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:02.653030 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s" podUID="9a7a8380-cdbe-4204-a374-8d42f30b87d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 02 01:29:02.786777 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:02.784809 4427 patch_prober.go:29] interesting pod/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 02 01:29:02.786777 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:02.784867 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c" podUID="c37ba466-a830-4246-832b-a8a4d0b5ab9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 02 01:29:02.805624 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:02.805491 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" podUID="47a9069f-1ffa-4c83-be8b-5e62f1d3275a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:02.998460 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:02.997495 4427 patch_prober.go:29] interesting pod/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 02 01:29:02.998460 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:02.997573 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4" podUID="9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 02 01:29:03.057910 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:03.057879 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" Nov 02 01:29:03.103741 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:03.103702 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" Nov 02 01:29:03.138126 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:03.138079 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" Nov 02 01:29:03.159427 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:03.157495 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" Nov 02 01:29:03.181402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:03.177521 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" podUID="eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:03.181402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:03.179842 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" podUID="706df1bd-19cf-4ab9-9aaa-3c12c97ccf45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 02 01:29:03.189583 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:03.187597 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" Nov 02 01:29:03.196408 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:03.195459 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" podUID="d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:03.249433 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:03.249323 4427 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 02 01:29:03.249678 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:03.249649 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb" podUID="917e0c7d-4eae-4d2d-bb59-f2de47fbfadb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 02 01:29:03.338412 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:03.328594 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" Nov 02 01:29:03.338412 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:03.330119 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" Nov 02 01:29:03.338412 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:03.335457 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" Nov 02 01:29:03.502316 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:03.502230 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" Nov 02 01:29:03.507016 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:03.505037 4427 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 02 01:29:03.507016 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:03.505103 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s" podUID="9a7a8380-cdbe-4204-a374-8d42f30b87d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 02 01:29:03.786011 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:03.784918 4427 patch_prober.go:29] interesting pod/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 02 01:29:03.786011 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:03.784989 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c" podUID="c37ba466-a830-4246-832b-a8a4d0b5ab9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 02 01:29:03.840848 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:03.840798 4427 patch_prober.go:29] interesting pod/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 02 01:29:03.841023 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:03.840869 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4" podUID="9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 02 01:29:04.692630 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:04.690715 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb" Nov 02 01:29:04.692630 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:04.692340 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb" Nov 02 01:29:04.705769 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:04.705673 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb" Nov 02 01:29:05.139128 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:05.139073 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s" Nov 02 01:29:05.141071 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:05.141049 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s" Nov 02 01:29:05.190445 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:05.187714 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4" Nov 02 01:29:05.190445 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:05.188722 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s" Nov 02 01:29:05.190445 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:05.189079 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4" Nov 02 01:29:05.234206 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:05.234173 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4" Nov 02 01:29:05.951326 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:05.951292 4427 patch_prober.go:29] interesting pod/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 01:29:05.967574 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:05.967537 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c" podUID="c37ba466-a830-4246-832b-a8a4d0b5ab9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:06.092151 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:06.092107 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" podUID="f67c954a-ce5b-41d9-8d99-7d126bd89d59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:06.282455 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:06.281480 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" podUID="366c78ee-4aaf-4b7c-84b4-17028538cb1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:06.304578 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:06.282708 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" podUID="cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:06.551456 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:06.537575 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" podUID="fb533b5f-3525-454a-8ce2-ea9246194f28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:06.551456 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:06.541474 4427 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" podUID="353a8a41-21ee-4470-861c-b53833beb106" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": context deadline exceeded" Nov 02 01:29:06.693886 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:06.685728 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c" Nov 02 01:29:06.693886 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:06.687336 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c" Nov 02 01:29:06.706569 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:06.706533 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c" Nov 02 01:29:06.928400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:06.916280 4427 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" podUID="f67c954a-ce5b-41d9-8d99-7d126bd89d59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": context deadline exceeded" Nov 02 01:29:06.975423 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:06.973042 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" podUID="9c056798-2cf8-47b1-9829-67cebda7a248" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:07.006438 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:06.994549 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" podUID="eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:07.094661 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:07.094622 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" podUID="f67c954a-ce5b-41d9-8d99-7d126bd89d59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:07.111467 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:07.095438 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" podUID="d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:07.113081 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:07.111729 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" Nov 02 01:29:07.131141 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:07.130456 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" podUID="353a8a41-21ee-4470-861c-b53833beb106" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:07.144319 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:07.141785 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" podUID="706df1bd-19cf-4ab9-9aaa-3c12c97ccf45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:07.189459 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:07.172737 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" podUID="c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:07.419099 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:07.410688 4427 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" podUID="cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": context deadline exceeded" Nov 02 01:29:07.419099 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:07.412272 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" podUID="7085c0ab-7ee3-4665-abad-2977c25f68ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Nov 02 01:29:07.438769 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:07.431287 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" podUID="cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 02 01:29:07.438769 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:07.432075 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" Nov 02 01:29:07.710576 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:07.710542 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" Nov 02 01:29:07.726411 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:07.722339 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" podUID="ea61971c-5a72-4609-b5a0-58a74b324bb7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:07.891933 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:07.891903 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" Nov 02 01:29:08.008596 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:08.008497 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" podUID="eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:08.010000 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:08.009976 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" Nov 02 01:29:08.092649 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:08.092600 4427 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" podUID="fb533b5f-3525-454a-8ce2-ea9246194f28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:08.164498 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:08.164442 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" podUID="706df1bd-19cf-4ab9-9aaa-3c12c97ccf45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:08.167655 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:08.167629 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" Nov 02 01:29:08.271167 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:08.271074 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" Nov 02 01:29:08.445405 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:08.436958 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" podUID="366c78ee-4aaf-4b7c-84b4-17028538cb1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:08.491988 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:08.491943 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" podUID="7085c0ab-7ee3-4665-abad-2977c25f68ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:08.492505 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:08.492471 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" Nov 02 01:29:08.591732 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:08.591649 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" Nov 02 01:29:08.716805 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:08.716485 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" Nov 02 01:29:08.737412 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:08.725498 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" podUID="ea61971c-5a72-4609-b5a0-58a74b324bb7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 02 01:29:08.737412 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:08.725871 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" Nov 02 01:29:08.916589 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:08.916560 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" Nov 02 01:29:24.725521 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:24.725483 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7"] Nov 02 01:29:24.809832 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:24.809797 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7" Nov 02 01:29:24.941747 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:24.941654 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7"] Nov 02 01:29:24.990462 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:24.970972 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3730ee2-6ba0-4dec-b103-214881818132-istiod-ca-cert\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7\" (UID: \"f3730ee2-6ba0-4dec-b103-214881818132\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7" Nov 02 01:29:24.990462 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:24.971046 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3730ee2-6ba0-4dec-b103-214881818132-workload-certs\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7\" (UID: \"f3730ee2-6ba0-4dec-b103-214881818132\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7" Nov 02 01:29:24.990462 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:24.971089 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3730ee2-6ba0-4dec-b103-214881818132-istio-podinfo\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7\" (UID: \"f3730ee2-6ba0-4dec-b103-214881818132\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7" Nov 02 01:29:24.990462 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:24.971144 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3730ee2-6ba0-4dec-b103-214881818132-istio-data\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7\" (UID: \"f3730ee2-6ba0-4dec-b103-214881818132\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7" Nov 02 01:29:24.990462 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:24.971201 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxq28\" (UniqueName: \"kubernetes.io/projected/f3730ee2-6ba0-4dec-b103-214881818132-kube-api-access-qxq28\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7\" (UID: \"f3730ee2-6ba0-4dec-b103-214881818132\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7" Nov 02 01:29:24.990462 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:24.971251 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3730ee2-6ba0-4dec-b103-214881818132-credential-socket\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7\" (UID: \"f3730ee2-6ba0-4dec-b103-214881818132\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7" Nov 02 01:29:24.990462 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:24.971421 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3730ee2-6ba0-4dec-b103-214881818132-istio-token\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7\" (UID: \"f3730ee2-6ba0-4dec-b103-214881818132\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7" Nov 02 01:29:24.990462 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:24.971472 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3730ee2-6ba0-4dec-b103-214881818132-workload-socket\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7\" (UID: \"f3730ee2-6ba0-4dec-b103-214881818132\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7" Nov 02 01:29:24.990462 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:24.971499 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3730ee2-6ba0-4dec-b103-214881818132-istio-envoy\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7\" (UID: \"f3730ee2-6ba0-4dec-b103-214881818132\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7" Nov 02 01:29:24.990462 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:24.988467 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" podUID="9c056798-2cf8-47b1-9829-67cebda7a248" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:25.031511 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:25.031461 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" podUID="d73ae484-a9b8-4565-a346-0d803a1981ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": context deadline exceeded" Nov 02 01:29:25.072505 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:25.072467 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3730ee2-6ba0-4dec-b103-214881818132-istio-token\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7\" (UID: \"f3730ee2-6ba0-4dec-b103-214881818132\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7" Nov 02 01:29:25.074007 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:25.073979 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3730ee2-6ba0-4dec-b103-214881818132-workload-socket\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7\" (UID: \"f3730ee2-6ba0-4dec-b103-214881818132\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7" Nov 02 01:29:25.089667 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:25.089637 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3730ee2-6ba0-4dec-b103-214881818132-istio-envoy\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7\" (UID: \"f3730ee2-6ba0-4dec-b103-214881818132\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7" Nov 02 01:29:25.089957 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:25.089936 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3730ee2-6ba0-4dec-b103-214881818132-istiod-ca-cert\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7\" (UID: \"f3730ee2-6ba0-4dec-b103-214881818132\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7" Nov 02 01:29:25.090123 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:25.090108 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3730ee2-6ba0-4dec-b103-214881818132-workload-certs\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7\" (UID: \"f3730ee2-6ba0-4dec-b103-214881818132\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7" Nov 02 01:29:25.090232 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:25.090219 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3730ee2-6ba0-4dec-b103-214881818132-istio-podinfo\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7\" (UID: \"f3730ee2-6ba0-4dec-b103-214881818132\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7" Nov 02 01:29:25.090397 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:25.090357 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3730ee2-6ba0-4dec-b103-214881818132-istio-data\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7\" (UID: \"f3730ee2-6ba0-4dec-b103-214881818132\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7" Nov 02 01:29:25.090583 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:25.090560 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxq28\" (UniqueName: \"kubernetes.io/projected/f3730ee2-6ba0-4dec-b103-214881818132-kube-api-access-qxq28\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7\" (UID: \"f3730ee2-6ba0-4dec-b103-214881818132\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7" Nov 02 01:29:25.090743 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:25.090726 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3730ee2-6ba0-4dec-b103-214881818132-credential-socket\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7\" (UID: \"f3730ee2-6ba0-4dec-b103-214881818132\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7" Nov 02 01:29:25.091051 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:25.091018 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" podUID="d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:25.091529 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:25.083835 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3730ee2-6ba0-4dec-b103-214881818132-workload-socket\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7\" (UID: \"f3730ee2-6ba0-4dec-b103-214881818132\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7" Nov 02 01:29:25.092921 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:25.092898 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3730ee2-6ba0-4dec-b103-214881818132-credential-socket\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7\" (UID: \"f3730ee2-6ba0-4dec-b103-214881818132\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7" Nov 02 01:29:25.093713 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:25.093690 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3730ee2-6ba0-4dec-b103-214881818132-istiod-ca-cert\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7\" (UID: \"f3730ee2-6ba0-4dec-b103-214881818132\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7" Nov 02 01:29:25.094109 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:25.094087 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3730ee2-6ba0-4dec-b103-214881818132-workload-certs\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7\" (UID: \"f3730ee2-6ba0-4dec-b103-214881818132\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7" Nov 02 01:29:25.101477 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:25.101450 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3730ee2-6ba0-4dec-b103-214881818132-istio-data\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7\" (UID: \"f3730ee2-6ba0-4dec-b103-214881818132\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7" Nov 02 01:29:25.108646 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:25.108610 4427 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" podUID="7085c0ab-7ee3-4665-abad-2977c25f68ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:25.110297 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:25.110244 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3730ee2-6ba0-4dec-b103-214881818132-istio-token\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7\" (UID: \"f3730ee2-6ba0-4dec-b103-214881818132\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7" Nov 02 01:29:25.133815 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:25.133386 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" podUID="353a8a41-21ee-4470-861c-b53833beb106" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:25.138397 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:25.134702 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3730ee2-6ba0-4dec-b103-214881818132-istio-envoy\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7\" (UID: \"f3730ee2-6ba0-4dec-b103-214881818132\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7" Nov 02 01:29:25.138397 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:25.135262 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3730ee2-6ba0-4dec-b103-214881818132-istio-podinfo\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7\" (UID: \"f3730ee2-6ba0-4dec-b103-214881818132\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7" Nov 02 01:29:25.154011 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:25.153978 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxq28\" (UniqueName: \"kubernetes.io/projected/f3730ee2-6ba0-4dec-b103-214881818132-kube-api-access-qxq28\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7\" (UID: \"f3730ee2-6ba0-4dec-b103-214881818132\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7" Nov 02 01:29:25.154319 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:25.154277 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" podUID="c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:25.208916 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:25.208648 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7" Nov 02 01:29:25.288567 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:25.288471 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" podUID="eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:25.302492 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:25.302451 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" podUID="366c78ee-4aaf-4b7c-84b4-17028538cb1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:25.403136 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:25.402414 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" podUID="fb533b5f-3525-454a-8ce2-ea9246194f28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:25.663360 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:25.663317 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" podUID="7085c0ab-7ee3-4665-abad-2977c25f68ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:25.722259 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:25.722210 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" podUID="706df1bd-19cf-4ab9-9aaa-3c12c97ccf45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:25.768178 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:25.764537 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" podUID="cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:25.842224 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:25.842154 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" podUID="47a9069f-1ffa-4c83-be8b-5e62f1d3275a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:26.043749 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:26.030480 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" podUID="f67c954a-ce5b-41d9-8d99-7d126bd89d59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 02 01:29:26.070782 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:26.070740 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" podUID="9c056798-2cf8-47b1-9829-67cebda7a248" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:26.071598 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:26.071571 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" Nov 02 01:29:26.076828 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:26.076451 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" podUID="ea61971c-5a72-4609-b5a0-58a74b324bb7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:26.092655 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:26.092594 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" podUID="d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Nov 02 01:29:26.093752 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:26.093730 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" Nov 02 01:29:26.147314 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:26.136237 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" podUID="353a8a41-21ee-4470-861c-b53833beb106" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 02 01:29:26.147314 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:26.146242 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" Nov 02 01:29:26.165517 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:26.164726 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" podUID="c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:26.167578 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:26.166196 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" Nov 02 01:29:26.334020 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:26.333697 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" Nov 02 01:29:26.409542 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:26.409500 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" podUID="fb533b5f-3525-454a-8ce2-ea9246194f28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:26.410608 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:26.410474 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" Nov 02 01:29:26.429695 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:26.419963 4427 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" podUID="353a8a41-21ee-4470-861c-b53833beb106" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:26.601944 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:26.601654 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7"] Nov 02 01:29:26.636586 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:26.636227 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" Nov 02 01:29:26.667941 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:26.667890 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" podUID="7085c0ab-7ee3-4665-abad-2977c25f68ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:26.668578 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:26.668554 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" Nov 02 01:29:26.729529 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:26.729486 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" podUID="706df1bd-19cf-4ab9-9aaa-3c12c97ccf45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 02 01:29:26.734536 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:26.734509 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" Nov 02 01:29:26.755212 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:26.751911 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:29:26.755212 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:26.752007 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:29:26.755212 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:26.752063 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:29:26.874350 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:26.845521 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" podUID="47a9069f-1ffa-4c83-be8b-5e62f1d3275a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:26.874350 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:26.845945 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" Nov 02 01:29:26.941104 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:26.939934 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" Nov 02 01:29:26.971513 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:26.971331 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" Nov 02 01:29:27.109206 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:27.107143 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" podUID="ea61971c-5a72-4609-b5a0-58a74b324bb7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:27.109206 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:27.108526 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" podUID="9c056798-2cf8-47b1-9829-67cebda7a248" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 01:29:27.110407 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:27.109575 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" Nov 02 01:29:27.118408 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:27.113883 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" podUID="f67c954a-ce5b-41d9-8d99-7d126bd89d59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:27.118408 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:27.114142 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" podUID="d73ae484-a9b8-4565-a346-0d803a1981ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": context deadline exceeded" Nov 02 01:29:27.118408 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:27.114402 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" Nov 02 01:29:27.166865 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:27.166819 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" podUID="353a8a41-21ee-4470-861c-b53833beb106" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:27.186693 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:27.186656 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" podUID="c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:27.229437 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:27.229409 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" Nov 02 01:29:27.313899 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:27.297450 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" podUID="366c78ee-4aaf-4b7c-84b4-17028538cb1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 02 01:29:27.313899 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:27.300347 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" Nov 02 01:29:27.335138 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:27.333350 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" Nov 02 01:29:27.335138 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:27.334872 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" Nov 02 01:29:27.351523 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:27.351421 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" Nov 02 01:29:27.388796 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:27.388723 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" Nov 02 01:29:27.665952 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:27.665862 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7" event={"ID":"f3730ee2-6ba0-4dec-b103-214881818132","Type":"ContainerStarted","Data":"a828e3e38efe8596a7047f188634226964ed3fd28db5919d0ae3846282c6744c"} Nov 02 01:29:27.666173 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:27.666156 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7" event={"ID":"f3730ee2-6ba0-4dec-b103-214881818132","Type":"ContainerStarted","Data":"ef4c3729061f1c2c96c8eac6d465f5f17b0388216823b2b50c66464b9148cdb2"} Nov 02 01:29:27.691983 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:27.691930 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7" podStartSLOduration=3.691917708 podStartE2EDuration="3.691917708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:29:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:29:27.689567158 +0000 UTC m=+872.310572370" watchObservedRunningTime="2026-11-02 01:29:27.691917708 +0000 UTC m=+872.312922919" Nov 02 01:29:28.209724 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:28.209686 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7" Nov 02 01:29:28.210880 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:28.210850 4427 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 02 01:29:28.211006 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:28.210899 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7" podUID="f3730ee2-6ba0-4dec-b103-214881818132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 02 01:29:28.722879 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:28.722844 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e26f5c63-8khb5"] Nov 02 01:29:28.726939 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:28.726922 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e26f5c63-8khb5" Nov 02 01:29:28.736301 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:28.736277 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 01:29:28.736562 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:28.736548 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\"" Nov 02 01:29:28.741039 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:28.741016 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e26f5c63-8khb5"] Nov 02 01:29:28.825663 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:28.825631 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf7743f7-236d-4094-ae86-08fac541fd80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e26f5c63-8khb5\" (UID: \"bf7743f7-236d-4094-ae86-08fac541fd80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e26f5c63-8khb5" Nov 02 01:29:28.825886 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:28.825698 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qncg\" (UniqueName: \"kubernetes.io/projected/bf7743f7-236d-4094-ae86-08fac541fd80-kube-api-access-4qncg\") pod \"maas-api-revoke-sub-e26f5c63-8khb5\" (UID: \"bf7743f7-236d-4094-ae86-08fac541fd80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e26f5c63-8khb5" Nov 02 01:29:28.825980 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:28.825872 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf7743f7-236d-4094-ae86-08fac541fd80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e26f5c63-8khb5\" (UID: \"bf7743f7-236d-4094-ae86-08fac541fd80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e26f5c63-8khb5" Nov 02 01:29:28.927280 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:28.927245 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf7743f7-236d-4094-ae86-08fac541fd80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e26f5c63-8khb5\" (UID: \"bf7743f7-236d-4094-ae86-08fac541fd80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e26f5c63-8khb5" Nov 02 01:29:28.927551 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:28.927299 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qncg\" (UniqueName: \"kubernetes.io/projected/bf7743f7-236d-4094-ae86-08fac541fd80-kube-api-access-4qncg\") pod \"maas-api-revoke-sub-e26f5c63-8khb5\" (UID: \"bf7743f7-236d-4094-ae86-08fac541fd80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e26f5c63-8khb5" Nov 02 01:29:28.927551 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:28.927331 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf7743f7-236d-4094-ae86-08fac541fd80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e26f5c63-8khb5\" (UID: \"bf7743f7-236d-4094-ae86-08fac541fd80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e26f5c63-8khb5" Nov 02 01:29:28.928004 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:28.927976 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf7743f7-236d-4094-ae86-08fac541fd80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e26f5c63-8khb5\" (UID: \"bf7743f7-236d-4094-ae86-08fac541fd80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e26f5c63-8khb5" Nov 02 01:29:28.930203 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:28.930179 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf7743f7-236d-4094-ae86-08fac541fd80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e26f5c63-8khb5\" (UID: \"bf7743f7-236d-4094-ae86-08fac541fd80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e26f5c63-8khb5" Nov 02 01:29:28.939657 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:28.939635 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qncg\" (UniqueName: \"kubernetes.io/projected/bf7743f7-236d-4094-ae86-08fac541fd80-kube-api-access-4qncg\") pod \"maas-api-revoke-sub-e26f5c63-8khb5\" (UID: \"bf7743f7-236d-4094-ae86-08fac541fd80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e26f5c63-8khb5" Nov 02 01:29:29.045288 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:29.045210 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e26f5c63-8khb5" Nov 02 01:29:29.210512 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:29.210469 4427 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 02 01:29:29.210981 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:29.210539 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7" podUID="f3730ee2-6ba0-4dec-b103-214881818132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 02 01:29:29.396120 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:29.396088 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e26f5c63-8khb5"] Nov 02 01:29:29.697487 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:29.697407 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e26f5c63-8khb5" event={"ID":"bf7743f7-236d-4094-ae86-08fac541fd80","Type":"ContainerStarted","Data":"3c7113d96600d15c3cc6c3bfbe33d039810055fce7f9c637a972b300f333b682"} Nov 02 01:29:30.210496 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:30.210458 4427 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 02 01:29:30.210715 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:30.210515 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7" podUID="f3730ee2-6ba0-4dec-b103-214881818132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 02 01:29:30.705919 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:30.705887 4427 generic.go:356] "Generic (PLEG): container finished" podID="bf7743f7-236d-4094-ae86-08fac541fd80" containerID="46a3de4207a3cb4ff259ece81dbfb31bfa1e20089e4fcb9174499515c59c7fec" exitCode=0 Nov 02 01:29:30.706100 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:30.705959 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e26f5c63-8khb5" event={"ID":"bf7743f7-236d-4094-ae86-08fac541fd80","Type":"ContainerDied","Data":"46a3de4207a3cb4ff259ece81dbfb31bfa1e20089e4fcb9174499515c59c7fec"} Nov 02 01:29:31.209666 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:31.209636 4427 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 02 01:29:31.209831 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:31.209685 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7" podUID="f3730ee2-6ba0-4dec-b103-214881818132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 02 01:29:31.610565 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:29:31.610531 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:29:31.955475 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:31.955446 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e26f5c63-8khb5" Nov 02 01:29:32.035043 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:32.035007 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf7743f7-236d-4094-ae86-08fac541fd80-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf7743f7-236d-4094-ae86-08fac541fd80-maas-api-openshift-ca\") pod \"bf7743f7-236d-4094-ae86-08fac541fd80\" (UID: \"bf7743f7-236d-4094-ae86-08fac541fd80\") " Nov 02 01:29:32.035289 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:32.035240 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf7743f7-236d-4094-ae86-08fac541fd80-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf7743f7-236d-4094-ae86-08fac541fd80-maas-api-xks-ca\") pod \"bf7743f7-236d-4094-ae86-08fac541fd80\" (UID: \"bf7743f7-236d-4094-ae86-08fac541fd80\") " Nov 02 01:29:32.035418 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:32.035299 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf7743f7-236d-4094-ae86-08fac541fd80-kube-api-access-4qncg\" (UniqueName: \"kubernetes.io/projected/bf7743f7-236d-4094-ae86-08fac541fd80-kube-api-access-4qncg\") pod \"bf7743f7-236d-4094-ae86-08fac541fd80\" (UID: \"bf7743f7-236d-4094-ae86-08fac541fd80\") " Nov 02 01:29:32.035673 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:32.035643 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf7743f7-236d-4094-ae86-08fac541fd80-maas-api-openshift-ca" pod "bf7743f7-236d-4094-ae86-08fac541fd80" (UID: "bf7743f7-236d-4094-ae86-08fac541fd80"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:29:32.036009 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:32.035983 4427 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf7743f7-236d-4094-ae86-08fac541fd80-maas-api-openshift-ca\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:29:32.038159 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:32.038130 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf7743f7-236d-4094-ae86-08fac541fd80-kube-api-access-4qncg" pod "bf7743f7-236d-4094-ae86-08fac541fd80" (UID: "bf7743f7-236d-4094-ae86-08fac541fd80"). InnerVolumeSpecName "kube-api-access-4qncg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:29:32.041930 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:32.041902 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf7743f7-236d-4094-ae86-08fac541fd80-maas-api-xks-ca" pod "bf7743f7-236d-4094-ae86-08fac541fd80" (UID: "bf7743f7-236d-4094-ae86-08fac541fd80"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:29:32.137872 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:32.137838 4427 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf7743f7-236d-4094-ae86-08fac541fd80-maas-api-xks-ca\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:29:32.137872 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:32.137868 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qncg\" (UniqueName: \"kubernetes.io/projected/bf7743f7-236d-4094-ae86-08fac541fd80-kube-api-access-4qncg\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:29:32.209749 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:32.209647 4427 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 02 01:29:32.209749 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:32.209714 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7" podUID="f3730ee2-6ba0-4dec-b103-214881818132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 02 01:29:32.723631 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:32.723595 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e26f5c63-8khb5" event={"ID":"bf7743f7-236d-4094-ae86-08fac541fd80","Type":"ContainerDied","Data":"3c7113d96600d15c3cc6c3bfbe33d039810055fce7f9c637a972b300f333b682"} Nov 02 01:29:32.724092 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:32.723655 4427 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c7113d96600d15c3cc6c3bfbe33d039810055fce7f9c637a972b300f333b682" Nov 02 01:29:32.724092 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:32.723612 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e26f5c63-8khb5" Nov 02 01:29:33.210122 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:33.210082 4427 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 02 01:29:33.210324 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:33.210149 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7" podUID="f3730ee2-6ba0-4dec-b103-214881818132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 02 01:29:34.209588 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:34.209556 4427 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 02 01:29:34.209993 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:34.209609 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7" podUID="f3730ee2-6ba0-4dec-b103-214881818132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 02 01:29:35.214541 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:35.214508 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7" Nov 02 01:29:35.215007 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:35.214987 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7" Nov 02 01:29:35.215774 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:35.215752 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7" Nov 02 01:29:39.073147 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.073099 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z"] Nov 02 01:29:39.073791 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.073776 4427 cpu_manager.go:414] "containerMap: removing container" podUID="bf7743f7-236d-4094-ae86-08fac541fd80" containerName="revoke-keys" Nov 02 01:29:39.073838 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.073793 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf7743f7-236d-4094-ae86-08fac541fd80" containerName="revoke-keys" Nov 02 01:29:39.073899 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.073890 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf7743f7-236d-4094-ae86-08fac541fd80" containerName="revoke-keys" Nov 02 01:29:39.076979 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.076960 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z" Nov 02 01:29:39.090313 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.090285 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z"] Nov 02 01:29:39.145782 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.145739 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/006dbe2b-932c-4c07-9430-51b9bd3b2bb2-istio-token\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z\" (UID: \"006dbe2b-932c-4c07-9430-51b9bd3b2bb2\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z" Nov 02 01:29:39.145981 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.145790 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/006dbe2b-932c-4c07-9430-51b9bd3b2bb2-istio-envoy\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z\" (UID: \"006dbe2b-932c-4c07-9430-51b9bd3b2bb2\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z" Nov 02 01:29:39.145981 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.145879 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/006dbe2b-932c-4c07-9430-51b9bd3b2bb2-istio-data\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z\" (UID: \"006dbe2b-932c-4c07-9430-51b9bd3b2bb2\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z" Nov 02 01:29:39.145981 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.145975 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/006dbe2b-932c-4c07-9430-51b9bd3b2bb2-workload-socket\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z\" (UID: \"006dbe2b-932c-4c07-9430-51b9bd3b2bb2\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z" Nov 02 01:29:39.146181 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.146099 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/006dbe2b-932c-4c07-9430-51b9bd3b2bb2-workload-certs\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z\" (UID: \"006dbe2b-932c-4c07-9430-51b9bd3b2bb2\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z" Nov 02 01:29:39.146347 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.146248 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/006dbe2b-932c-4c07-9430-51b9bd3b2bb2-istio-podinfo\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z\" (UID: \"006dbe2b-932c-4c07-9430-51b9bd3b2bb2\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z" Nov 02 01:29:39.146652 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.146634 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/006dbe2b-932c-4c07-9430-51b9bd3b2bb2-credential-socket\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z\" (UID: \"006dbe2b-932c-4c07-9430-51b9bd3b2bb2\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z" Nov 02 01:29:39.146774 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.146760 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92d8g\" (UniqueName: \"kubernetes.io/projected/006dbe2b-932c-4c07-9430-51b9bd3b2bb2-kube-api-access-92d8g\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z\" (UID: \"006dbe2b-932c-4c07-9430-51b9bd3b2bb2\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z" Nov 02 01:29:39.146824 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.146788 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/006dbe2b-932c-4c07-9430-51b9bd3b2bb2-istiod-ca-cert\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z\" (UID: \"006dbe2b-932c-4c07-9430-51b9bd3b2bb2\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z" Nov 02 01:29:39.248009 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.247974 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/006dbe2b-932c-4c07-9430-51b9bd3b2bb2-workload-socket\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z\" (UID: \"006dbe2b-932c-4c07-9430-51b9bd3b2bb2\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z" Nov 02 01:29:39.248197 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.248080 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/006dbe2b-932c-4c07-9430-51b9bd3b2bb2-workload-certs\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z\" (UID: \"006dbe2b-932c-4c07-9430-51b9bd3b2bb2\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z" Nov 02 01:29:39.248197 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.248110 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/006dbe2b-932c-4c07-9430-51b9bd3b2bb2-istio-podinfo\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z\" (UID: \"006dbe2b-932c-4c07-9430-51b9bd3b2bb2\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z" Nov 02 01:29:39.248197 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.248138 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/006dbe2b-932c-4c07-9430-51b9bd3b2bb2-credential-socket\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z\" (UID: \"006dbe2b-932c-4c07-9430-51b9bd3b2bb2\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z" Nov 02 01:29:39.248357 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.248260 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92d8g\" (UniqueName: \"kubernetes.io/projected/006dbe2b-932c-4c07-9430-51b9bd3b2bb2-kube-api-access-92d8g\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z\" (UID: \"006dbe2b-932c-4c07-9430-51b9bd3b2bb2\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z" Nov 02 01:29:39.248357 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.248300 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/006dbe2b-932c-4c07-9430-51b9bd3b2bb2-istiod-ca-cert\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z\" (UID: \"006dbe2b-932c-4c07-9430-51b9bd3b2bb2\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z" Nov 02 01:29:39.248506 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.248439 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/006dbe2b-932c-4c07-9430-51b9bd3b2bb2-istio-token\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z\" (UID: \"006dbe2b-932c-4c07-9430-51b9bd3b2bb2\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z" Nov 02 01:29:39.248506 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.248479 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/006dbe2b-932c-4c07-9430-51b9bd3b2bb2-workload-socket\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z\" (UID: \"006dbe2b-932c-4c07-9430-51b9bd3b2bb2\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z" Nov 02 01:29:39.248614 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.248517 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/006dbe2b-932c-4c07-9430-51b9bd3b2bb2-workload-certs\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z\" (UID: \"006dbe2b-932c-4c07-9430-51b9bd3b2bb2\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z" Nov 02 01:29:39.248614 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.248598 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/006dbe2b-932c-4c07-9430-51b9bd3b2bb2-credential-socket\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z\" (UID: \"006dbe2b-932c-4c07-9430-51b9bd3b2bb2\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z" Nov 02 01:29:39.248697 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.248670 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/006dbe2b-932c-4c07-9430-51b9bd3b2bb2-istio-envoy\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z\" (UID: \"006dbe2b-932c-4c07-9430-51b9bd3b2bb2\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z" Nov 02 01:29:39.248758 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.248712 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/006dbe2b-932c-4c07-9430-51b9bd3b2bb2-istio-data\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z\" (UID: \"006dbe2b-932c-4c07-9430-51b9bd3b2bb2\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z" Nov 02 01:29:39.249036 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.249011 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/006dbe2b-932c-4c07-9430-51b9bd3b2bb2-istio-data\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z\" (UID: \"006dbe2b-932c-4c07-9430-51b9bd3b2bb2\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z" Nov 02 01:29:39.249165 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.249067 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/006dbe2b-932c-4c07-9430-51b9bd3b2bb2-istiod-ca-cert\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z\" (UID: \"006dbe2b-932c-4c07-9430-51b9bd3b2bb2\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z" Nov 02 01:29:39.250778 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.250753 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/006dbe2b-932c-4c07-9430-51b9bd3b2bb2-istio-podinfo\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z\" (UID: \"006dbe2b-932c-4c07-9430-51b9bd3b2bb2\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z" Nov 02 01:29:39.250778 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.250771 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/006dbe2b-932c-4c07-9430-51b9bd3b2bb2-istio-envoy\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z\" (UID: \"006dbe2b-932c-4c07-9430-51b9bd3b2bb2\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z" Nov 02 01:29:39.262075 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.262051 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/006dbe2b-932c-4c07-9430-51b9bd3b2bb2-istio-token\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z\" (UID: \"006dbe2b-932c-4c07-9430-51b9bd3b2bb2\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z" Nov 02 01:29:39.262214 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.262136 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92d8g\" (UniqueName: \"kubernetes.io/projected/006dbe2b-932c-4c07-9430-51b9bd3b2bb2-kube-api-access-92d8g\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z\" (UID: \"006dbe2b-932c-4c07-9430-51b9bd3b2bb2\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z" Nov 02 01:29:39.392741 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.392711 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z" Nov 02 01:29:39.515186 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.515149 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2"] Nov 02 01:29:39.521922 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.521895 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2" Nov 02 01:29:39.542670 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.542643 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2"] Nov 02 01:29:39.549098 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.549065 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv"] Nov 02 01:29:39.559959 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.559287 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv" Nov 02 01:29:39.562962 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.562932 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv"] Nov 02 01:29:39.660284 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.660203 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hng4n\" (UniqueName: \"kubernetes.io/projected/d5cb7660-e2aa-48a2-9eac-508cd83f93cb-kube-api-access-hng4n\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv\" (UID: \"d5cb7660-e2aa-48a2-9eac-508cd83f93cb\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv" Nov 02 01:29:39.660284 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.660244 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1a145f0-821a-4d90-a373-fd65c74c8e57-istio-data\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2\" (UID: \"e1a145f0-821a-4d90-a373-fd65c74c8e57\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2" Nov 02 01:29:39.660547 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.660395 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck6xq\" (UniqueName: \"kubernetes.io/projected/e1a145f0-821a-4d90-a373-fd65c74c8e57-kube-api-access-ck6xq\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2\" (UID: \"e1a145f0-821a-4d90-a373-fd65c74c8e57\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2" Nov 02 01:29:39.660547 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.660499 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1a145f0-821a-4d90-a373-fd65c74c8e57-workload-certs\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2\" (UID: \"e1a145f0-821a-4d90-a373-fd65c74c8e57\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2" Nov 02 01:29:39.660665 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.660647 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1a145f0-821a-4d90-a373-fd65c74c8e57-istiod-ca-cert\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2\" (UID: \"e1a145f0-821a-4d90-a373-fd65c74c8e57\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2" Nov 02 01:29:39.660855 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.660828 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5cb7660-e2aa-48a2-9eac-508cd83f93cb-istiod-ca-cert\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv\" (UID: \"d5cb7660-e2aa-48a2-9eac-508cd83f93cb\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv" Nov 02 01:29:39.660916 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.660893 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1a145f0-821a-4d90-a373-fd65c74c8e57-credential-socket\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2\" (UID: \"e1a145f0-821a-4d90-a373-fd65c74c8e57\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2" Nov 02 01:29:39.660992 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.660955 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5cb7660-e2aa-48a2-9eac-508cd83f93cb-istio-data\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv\" (UID: \"d5cb7660-e2aa-48a2-9eac-508cd83f93cb\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv" Nov 02 01:29:39.661125 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.661059 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1a145f0-821a-4d90-a373-fd65c74c8e57-workload-socket\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2\" (UID: \"e1a145f0-821a-4d90-a373-fd65c74c8e57\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2" Nov 02 01:29:39.661125 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.661114 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1a145f0-821a-4d90-a373-fd65c74c8e57-istio-token\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2\" (UID: \"e1a145f0-821a-4d90-a373-fd65c74c8e57\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2" Nov 02 01:29:39.661209 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.661190 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1a145f0-821a-4d90-a373-fd65c74c8e57-istio-envoy\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2\" (UID: \"e1a145f0-821a-4d90-a373-fd65c74c8e57\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2" Nov 02 01:29:39.661249 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.661224 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5cb7660-e2aa-48a2-9eac-508cd83f93cb-istio-envoy\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv\" (UID: \"d5cb7660-e2aa-48a2-9eac-508cd83f93cb\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv" Nov 02 01:29:39.661299 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.661282 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5cb7660-e2aa-48a2-9eac-508cd83f93cb-workload-certs\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv\" (UID: \"d5cb7660-e2aa-48a2-9eac-508cd83f93cb\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv" Nov 02 01:29:39.661345 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.661338 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5cb7660-e2aa-48a2-9eac-508cd83f93cb-workload-socket\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv\" (UID: \"d5cb7660-e2aa-48a2-9eac-508cd83f93cb\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv" Nov 02 01:29:39.661438 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.661367 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5cb7660-e2aa-48a2-9eac-508cd83f93cb-credential-socket\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv\" (UID: \"d5cb7660-e2aa-48a2-9eac-508cd83f93cb\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv" Nov 02 01:29:39.661438 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.661433 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1a145f0-821a-4d90-a373-fd65c74c8e57-istio-podinfo\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2\" (UID: \"e1a145f0-821a-4d90-a373-fd65c74c8e57\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2" Nov 02 01:29:39.661522 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.661493 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5cb7660-e2aa-48a2-9eac-508cd83f93cb-istio-token\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv\" (UID: \"d5cb7660-e2aa-48a2-9eac-508cd83f93cb\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv" Nov 02 01:29:39.661609 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.661590 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5cb7660-e2aa-48a2-9eac-508cd83f93cb-istio-podinfo\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv\" (UID: \"d5cb7660-e2aa-48a2-9eac-508cd83f93cb\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv" Nov 02 01:29:39.681148 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.681117 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z"] Nov 02 01:29:39.682226 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:29:39.682182 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod006dbe2b_932c_4c07_9430_51b9bd3b2bb2.slice/crio-6f852bde404675ef972aeae79f52bdc4e78ff731e8f2a38ddfc377ca5929e782 WatchSource:0}: Error finding container 6f852bde404675ef972aeae79f52bdc4e78ff731e8f2a38ddfc377ca5929e782: Status 404 returned error can't find the container with id 6f852bde404675ef972aeae79f52bdc4e78ff731e8f2a38ddfc377ca5929e782 Nov 02 01:29:39.685458 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.685416 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:29:39.685577 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.685510 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:29:39.685577 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.685556 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:29:39.763235 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.763208 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5cb7660-e2aa-48a2-9eac-508cd83f93cb-istio-token\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv\" (UID: \"d5cb7660-e2aa-48a2-9eac-508cd83f93cb\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv" Nov 02 01:29:39.763313 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.763256 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5cb7660-e2aa-48a2-9eac-508cd83f93cb-istio-podinfo\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv\" (UID: \"d5cb7660-e2aa-48a2-9eac-508cd83f93cb\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv" Nov 02 01:29:39.763313 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.763290 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hng4n\" (UniqueName: \"kubernetes.io/projected/d5cb7660-e2aa-48a2-9eac-508cd83f93cb-kube-api-access-hng4n\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv\" (UID: \"d5cb7660-e2aa-48a2-9eac-508cd83f93cb\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv" Nov 02 01:29:39.763313 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.763309 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1a145f0-821a-4d90-a373-fd65c74c8e57-istio-data\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2\" (UID: \"e1a145f0-821a-4d90-a373-fd65c74c8e57\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2" Nov 02 01:29:39.763483 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.763358 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck6xq\" (UniqueName: \"kubernetes.io/projected/e1a145f0-821a-4d90-a373-fd65c74c8e57-kube-api-access-ck6xq\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2\" (UID: \"e1a145f0-821a-4d90-a373-fd65c74c8e57\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2" Nov 02 01:29:39.763483 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.763428 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1a145f0-821a-4d90-a373-fd65c74c8e57-workload-certs\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2\" (UID: \"e1a145f0-821a-4d90-a373-fd65c74c8e57\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2" Nov 02 01:29:39.763767 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.763742 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1a145f0-821a-4d90-a373-fd65c74c8e57-istio-data\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2\" (UID: \"e1a145f0-821a-4d90-a373-fd65c74c8e57\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2" Nov 02 01:29:39.763855 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.763824 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1a145f0-821a-4d90-a373-fd65c74c8e57-istiod-ca-cert\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2\" (UID: \"e1a145f0-821a-4d90-a373-fd65c74c8e57\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2" Nov 02 01:29:39.763954 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.763940 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5cb7660-e2aa-48a2-9eac-508cd83f93cb-istiod-ca-cert\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv\" (UID: \"d5cb7660-e2aa-48a2-9eac-508cd83f93cb\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv" Nov 02 01:29:39.764014 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.763978 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1a145f0-821a-4d90-a373-fd65c74c8e57-credential-socket\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2\" (UID: \"e1a145f0-821a-4d90-a373-fd65c74c8e57\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2" Nov 02 01:29:39.764014 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.764004 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5cb7660-e2aa-48a2-9eac-508cd83f93cb-istio-data\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv\" (UID: \"d5cb7660-e2aa-48a2-9eac-508cd83f93cb\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv" Nov 02 01:29:39.764116 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.764049 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1a145f0-821a-4d90-a373-fd65c74c8e57-workload-socket\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2\" (UID: \"e1a145f0-821a-4d90-a373-fd65c74c8e57\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2" Nov 02 01:29:39.764116 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.764083 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1a145f0-821a-4d90-a373-fd65c74c8e57-workload-certs\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2\" (UID: \"e1a145f0-821a-4d90-a373-fd65c74c8e57\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2" Nov 02 01:29:39.764223 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.764115 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1a145f0-821a-4d90-a373-fd65c74c8e57-istio-token\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2\" (UID: \"e1a145f0-821a-4d90-a373-fd65c74c8e57\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2" Nov 02 01:29:39.764223 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.764149 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1a145f0-821a-4d90-a373-fd65c74c8e57-istio-envoy\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2\" (UID: \"e1a145f0-821a-4d90-a373-fd65c74c8e57\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2" Nov 02 01:29:39.764223 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.764174 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5cb7660-e2aa-48a2-9eac-508cd83f93cb-istio-envoy\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv\" (UID: \"d5cb7660-e2aa-48a2-9eac-508cd83f93cb\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv" Nov 02 01:29:39.764223 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.764202 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5cb7660-e2aa-48a2-9eac-508cd83f93cb-workload-certs\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv\" (UID: \"d5cb7660-e2aa-48a2-9eac-508cd83f93cb\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv" Nov 02 01:29:39.764535 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.764263 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5cb7660-e2aa-48a2-9eac-508cd83f93cb-workload-socket\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv\" (UID: \"d5cb7660-e2aa-48a2-9eac-508cd83f93cb\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv" Nov 02 01:29:39.764535 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.764311 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5cb7660-e2aa-48a2-9eac-508cd83f93cb-credential-socket\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv\" (UID: \"d5cb7660-e2aa-48a2-9eac-508cd83f93cb\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv" Nov 02 01:29:39.764535 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.764350 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1a145f0-821a-4d90-a373-fd65c74c8e57-istio-podinfo\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2\" (UID: \"e1a145f0-821a-4d90-a373-fd65c74c8e57\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2" Nov 02 01:29:39.764535 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.764449 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1a145f0-821a-4d90-a373-fd65c74c8e57-credential-socket\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2\" (UID: \"e1a145f0-821a-4d90-a373-fd65c74c8e57\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2" Nov 02 01:29:39.764740 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.764593 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1a145f0-821a-4d90-a373-fd65c74c8e57-istiod-ca-cert\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2\" (UID: \"e1a145f0-821a-4d90-a373-fd65c74c8e57\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2" Nov 02 01:29:39.764740 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.764636 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5cb7660-e2aa-48a2-9eac-508cd83f93cb-istio-data\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv\" (UID: \"d5cb7660-e2aa-48a2-9eac-508cd83f93cb\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv" Nov 02 01:29:39.764913 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.764882 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1a145f0-821a-4d90-a373-fd65c74c8e57-workload-socket\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2\" (UID: \"e1a145f0-821a-4d90-a373-fd65c74c8e57\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2" Nov 02 01:29:39.764913 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.764900 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5cb7660-e2aa-48a2-9eac-508cd83f93cb-istiod-ca-cert\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv\" (UID: \"d5cb7660-e2aa-48a2-9eac-508cd83f93cb\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv" Nov 02 01:29:39.765429 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.765406 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5cb7660-e2aa-48a2-9eac-508cd83f93cb-workload-certs\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv\" (UID: \"d5cb7660-e2aa-48a2-9eac-508cd83f93cb\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv" Nov 02 01:29:39.765603 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.765577 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5cb7660-e2aa-48a2-9eac-508cd83f93cb-credential-socket\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv\" (UID: \"d5cb7660-e2aa-48a2-9eac-508cd83f93cb\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv" Nov 02 01:29:39.765847 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.765631 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5cb7660-e2aa-48a2-9eac-508cd83f93cb-workload-socket\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv\" (UID: \"d5cb7660-e2aa-48a2-9eac-508cd83f93cb\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv" Nov 02 01:29:39.766556 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.766529 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5cb7660-e2aa-48a2-9eac-508cd83f93cb-istio-podinfo\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv\" (UID: \"d5cb7660-e2aa-48a2-9eac-508cd83f93cb\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv" Nov 02 01:29:39.767149 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.767130 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5cb7660-e2aa-48a2-9eac-508cd83f93cb-istio-envoy\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv\" (UID: \"d5cb7660-e2aa-48a2-9eac-508cd83f93cb\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv" Nov 02 01:29:39.767302 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.767285 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1a145f0-821a-4d90-a373-fd65c74c8e57-istio-envoy\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2\" (UID: \"e1a145f0-821a-4d90-a373-fd65c74c8e57\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2" Nov 02 01:29:39.768469 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.768444 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1a145f0-821a-4d90-a373-fd65c74c8e57-istio-podinfo\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2\" (UID: \"e1a145f0-821a-4d90-a373-fd65c74c8e57\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2" Nov 02 01:29:39.777053 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.777023 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5cb7660-e2aa-48a2-9eac-508cd83f93cb-istio-token\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv\" (UID: \"d5cb7660-e2aa-48a2-9eac-508cd83f93cb\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv" Nov 02 01:29:39.777162 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.777150 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hng4n\" (UniqueName: \"kubernetes.io/projected/d5cb7660-e2aa-48a2-9eac-508cd83f93cb-kube-api-access-hng4n\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv\" (UID: \"d5cb7660-e2aa-48a2-9eac-508cd83f93cb\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv" Nov 02 01:29:39.777220 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.777195 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z" event={"ID":"006dbe2b-932c-4c07-9430-51b9bd3b2bb2","Type":"ContainerStarted","Data":"dd4a4eb0c94ba307188bf1606d6451ec807a34ce04062d0daa26a941bfd1bb47"} Nov 02 01:29:39.777307 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.777289 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z" event={"ID":"006dbe2b-932c-4c07-9430-51b9bd3b2bb2","Type":"ContainerStarted","Data":"6f852bde404675ef972aeae79f52bdc4e78ff731e8f2a38ddfc377ca5929e782"} Nov 02 01:29:39.777474 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.777459 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1a145f0-821a-4d90-a373-fd65c74c8e57-istio-token\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2\" (UID: \"e1a145f0-821a-4d90-a373-fd65c74c8e57\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2" Nov 02 01:29:39.777474 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.777469 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck6xq\" (UniqueName: \"kubernetes.io/projected/e1a145f0-821a-4d90-a373-fd65c74c8e57-kube-api-access-ck6xq\") pod \"e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2\" (UID: \"e1a145f0-821a-4d90-a373-fd65c74c8e57\") " pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2" Nov 02 01:29:39.843177 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.843135 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2" Nov 02 01:29:39.879167 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:39.879136 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv" Nov 02 01:29:40.136085 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:29:40.136057 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1a145f0_821a_4d90_a373_fd65c74c8e57.slice/crio-7f82efd5152c8ce1f636b88165054cb318804c5f0d1a7f6c8cd20887a2d1e257 WatchSource:0}: Error finding container 7f82efd5152c8ce1f636b88165054cb318804c5f0d1a7f6c8cd20887a2d1e257: Status 404 returned error can't find the container with id 7f82efd5152c8ce1f636b88165054cb318804c5f0d1a7f6c8cd20887a2d1e257 Nov 02 01:29:40.137558 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:40.137532 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2"] Nov 02 01:29:40.138674 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:40.138642 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:29:40.138741 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:40.138709 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:29:40.138774 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:40.138740 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:29:40.204941 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:40.204914 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv"] Nov 02 01:29:40.216390 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:40.210549 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:29:40.216390 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:40.210638 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:29:40.216390 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:40.210689 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:29:40.811148 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:40.811117 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv" event={"ID":"d5cb7660-e2aa-48a2-9eac-508cd83f93cb","Type":"ContainerStarted","Data":"653247f1830371c147dfdeaf91a2a979e1c486b36c57a66759680b540f84909f"} Nov 02 01:29:40.811422 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:40.811403 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv" event={"ID":"d5cb7660-e2aa-48a2-9eac-508cd83f93cb","Type":"ContainerStarted","Data":"bb08b8059619f95033814157e5ed08abc5b1b254aff8a0aa7f7bf814ce89b128"} Nov 02 01:29:40.816276 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:40.816247 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2" event={"ID":"e1a145f0-821a-4d90-a373-fd65c74c8e57","Type":"ContainerStarted","Data":"97c5be5723ced963fd89d722494b3b29df47f8c81df7a475b1a5a3dac7b8656a"} Nov 02 01:29:40.816559 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:40.816532 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2" event={"ID":"e1a145f0-821a-4d90-a373-fd65c74c8e57","Type":"ContainerStarted","Data":"7f82efd5152c8ce1f636b88165054cb318804c5f0d1a7f6c8cd20887a2d1e257"} Nov 02 01:29:40.840802 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:40.840720 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv" podStartSLOduration=1.8406977169999998 podStartE2EDuration="1.840697717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:29:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:29:40.836795904 +0000 UTC m=+885.457801117" watchObservedRunningTime="2026-11-02 01:29:40.840697717 +0000 UTC m=+885.461702954" Nov 02 01:29:40.843528 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:40.843474 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2" Nov 02 01:29:40.882402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:40.881531 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv" Nov 02 01:29:40.903403 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:40.889938 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z" podStartSLOduration=1.889914165 podStartE2EDuration="1.889914165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:29:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:29:40.863009573 +0000 UTC m=+885.484014786" watchObservedRunningTime="2026-11-02 01:29:40.889914165 +0000 UTC m=+885.510919378" Nov 02 01:29:40.923893 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:40.916713 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2" podStartSLOduration=1.916693548 podStartE2EDuration="1.916693548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:29:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:29:40.912102747 +0000 UTC m=+885.533107968" watchObservedRunningTime="2026-11-02 01:29:40.916693548 +0000 UTC m=+885.537698761" Nov 02 01:29:41.400399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:41.399436 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z" Nov 02 01:29:41.400399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:41.399589 4427 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 02 01:29:41.400399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:41.399632 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z" podUID="006dbe2b-932c-4c07-9430-51b9bd3b2bb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 02 01:29:41.848309 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:41.847867 4427 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 02 01:29:41.848309 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:41.847984 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2" podUID="e1a145f0-821a-4d90-a373-fd65c74c8e57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 02 01:29:41.892402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:41.891550 4427 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 02 01:29:41.892402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:41.891616 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv" podUID="d5cb7660-e2aa-48a2-9eac-508cd83f93cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 02 01:29:42.403404 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:42.399885 4427 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 02 01:29:42.403404 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:42.399953 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z" podUID="006dbe2b-932c-4c07-9430-51b9bd3b2bb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 02 01:29:42.850479 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:42.850344 4427 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 02 01:29:42.850777 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:42.850748 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2" podUID="e1a145f0-821a-4d90-a373-fd65c74c8e57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 02 01:29:42.892403 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:42.879843 4427 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 02 01:29:42.892403 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:42.879922 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv" podUID="d5cb7660-e2aa-48a2-9eac-508cd83f93cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 02 01:29:43.448261 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:43.447541 4427 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 02 01:29:43.448261 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:43.447603 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z" podUID="006dbe2b-932c-4c07-9430-51b9bd3b2bb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 02 01:29:43.847314 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:43.847219 4427 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 02 01:29:43.847986 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:43.847955 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2" podUID="e1a145f0-821a-4d90-a373-fd65c74c8e57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 02 01:29:43.920401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:43.882325 4427 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 02 01:29:43.973521 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:43.973467 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv" podUID="d5cb7660-e2aa-48a2-9eac-508cd83f93cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 02 01:29:44.464686 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:44.462150 4427 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 02 01:29:44.464686 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:44.462225 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z" podUID="006dbe2b-932c-4c07-9430-51b9bd3b2bb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 02 01:29:44.936400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:44.910921 4427 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 02 01:29:44.947326 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:44.925555 4427 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 02 01:29:44.947326 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:44.947271 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2" podUID="e1a145f0-821a-4d90-a373-fd65c74c8e57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 02 01:29:44.948023 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:44.947993 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv" podUID="d5cb7660-e2aa-48a2-9eac-508cd83f93cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 02 01:29:45.418843 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:45.418805 4427 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 02 01:29:45.419164 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:45.419125 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z" podUID="006dbe2b-932c-4c07-9430-51b9bd3b2bb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 02 01:29:45.843895 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:45.843785 4427 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 02 01:29:45.844507 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:45.844472 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2" podUID="e1a145f0-821a-4d90-a373-fd65c74c8e57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 02 01:29:45.881404 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:45.880858 4427 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 02 01:29:45.881404 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:45.880938 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv" podUID="d5cb7660-e2aa-48a2-9eac-508cd83f93cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 02 01:29:46.402402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:46.393672 4427 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 02 01:29:46.402402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:46.393732 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z" podUID="006dbe2b-932c-4c07-9430-51b9bd3b2bb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 02 01:29:46.844872 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:46.844764 4427 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 02 01:29:46.844872 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:46.844828 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2" podUID="e1a145f0-821a-4d90-a373-fd65c74c8e57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 02 01:29:46.879958 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:46.879920 4427 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 02 01:29:46.880180 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:46.879999 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv" podUID="d5cb7660-e2aa-48a2-9eac-508cd83f93cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 02 01:29:47.886401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:47.885859 4427 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 02 01:29:47.886401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:47.885943 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv" podUID="d5cb7660-e2aa-48a2-9eac-508cd83f93cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 02 01:29:47.908397 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:47.901564 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z" Nov 02 01:29:47.908397 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:47.903493 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z" Nov 02 01:29:47.908397 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:47.905060 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z" Nov 02 01:29:48.234095 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:48.234012 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2" Nov 02 01:29:48.234972 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:48.234949 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2" Nov 02 01:29:48.235900 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:48.235745 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2" Nov 02 01:29:48.885092 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:48.885064 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv" Nov 02 01:29:48.885521 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:48.885501 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv" Nov 02 01:29:48.886237 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:48.886216 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv" Nov 02 01:29:49.780035 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:49.780000 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03a8ca0d-z4v8m"] Nov 02 01:29:49.784785 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:49.784759 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a8ca0d-z4v8m" Nov 02 01:29:49.791230 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:49.791194 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03a8ca0d-z4v8m"] Nov 02 01:29:49.791556 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:49.791533 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\"" Nov 02 01:29:49.791556 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:49.791556 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 01:29:49.832197 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:49.832150 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf676aff-c089-43c6-95de-45c72e2c22ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03a8ca0d-z4v8m\" (UID: \"bf676aff-c089-43c6-95de-45c72e2c22ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a8ca0d-z4v8m" Nov 02 01:29:49.832416 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:49.832318 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzxjm\" (UniqueName: \"kubernetes.io/projected/bf676aff-c089-43c6-95de-45c72e2c22ab-kube-api-access-bzxjm\") pod \"maas-api-revoke-sub-03a8ca0d-z4v8m\" (UID: \"bf676aff-c089-43c6-95de-45c72e2c22ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a8ca0d-z4v8m" Nov 02 01:29:49.832490 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:49.832448 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf676aff-c089-43c6-95de-45c72e2c22ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03a8ca0d-z4v8m\" (UID: \"bf676aff-c089-43c6-95de-45c72e2c22ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a8ca0d-z4v8m" Nov 02 01:29:49.935842 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:49.935740 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf676aff-c089-43c6-95de-45c72e2c22ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03a8ca0d-z4v8m\" (UID: \"bf676aff-c089-43c6-95de-45c72e2c22ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a8ca0d-z4v8m" Nov 02 01:29:49.936167 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:49.936142 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzxjm\" (UniqueName: \"kubernetes.io/projected/bf676aff-c089-43c6-95de-45c72e2c22ab-kube-api-access-bzxjm\") pod \"maas-api-revoke-sub-03a8ca0d-z4v8m\" (UID: \"bf676aff-c089-43c6-95de-45c72e2c22ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a8ca0d-z4v8m" Nov 02 01:29:49.936315 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:49.936300 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf676aff-c089-43c6-95de-45c72e2c22ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03a8ca0d-z4v8m\" (UID: \"bf676aff-c089-43c6-95de-45c72e2c22ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a8ca0d-z4v8m" Nov 02 01:29:49.945494 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:49.945461 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf676aff-c089-43c6-95de-45c72e2c22ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03a8ca0d-z4v8m\" (UID: \"bf676aff-c089-43c6-95de-45c72e2c22ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a8ca0d-z4v8m" Nov 02 01:29:49.948657 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:49.948629 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf676aff-c089-43c6-95de-45c72e2c22ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03a8ca0d-z4v8m\" (UID: \"bf676aff-c089-43c6-95de-45c72e2c22ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a8ca0d-z4v8m" Nov 02 01:29:49.958969 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:49.958934 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzxjm\" (UniqueName: \"kubernetes.io/projected/bf676aff-c089-43c6-95de-45c72e2c22ab-kube-api-access-bzxjm\") pod \"maas-api-revoke-sub-03a8ca0d-z4v8m\" (UID: \"bf676aff-c089-43c6-95de-45c72e2c22ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a8ca0d-z4v8m" Nov 02 01:29:50.102290 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:50.102210 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a8ca0d-z4v8m" Nov 02 01:29:50.393967 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:50.393939 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03a8ca0d-z4v8m"] Nov 02 01:29:51.278275 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:51.278223 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a8ca0d-z4v8m" event={"ID":"bf676aff-c089-43c6-95de-45c72e2c22ab","Type":"ContainerStarted","Data":"9a68797386cd091dff4b516815baa485f3cc08184f57cb45711379978cdad27c"} Nov 02 01:29:51.278860 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:51.278292 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a8ca0d-z4v8m" event={"ID":"bf676aff-c089-43c6-95de-45c72e2c22ab","Type":"ContainerStarted","Data":"d7f35d2551774ff13160d1aacdb7866780b3980f3eb8ceee29a6a9ef638c1260"} Nov 02 01:29:51.299482 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:51.299427 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a8ca0d-z4v8m" podStartSLOduration=2.299406401 podStartE2EDuration="2.299406401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:29:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:29:51.297482524 +0000 UTC m=+895.918487737" watchObservedRunningTime="2026-11-02 01:29:51.299406401 +0000 UTC m=+895.920411607" Nov 02 01:29:52.287382 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:52.287346 4427 generic.go:356] "Generic (PLEG): container finished" podID="bf676aff-c089-43c6-95de-45c72e2c22ab" containerID="9a68797386cd091dff4b516815baa485f3cc08184f57cb45711379978cdad27c" exitCode=0 Nov 02 01:29:52.287868 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:52.287418 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a8ca0d-z4v8m" event={"ID":"bf676aff-c089-43c6-95de-45c72e2c22ab","Type":"ContainerDied","Data":"9a68797386cd091dff4b516815baa485f3cc08184f57cb45711379978cdad27c"} Nov 02 01:29:53.464930 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:53.464901 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a8ca0d-z4v8m" Nov 02 01:29:53.599908 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:53.599825 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf676aff-c089-43c6-95de-45c72e2c22ab-kube-api-access-bzxjm\" (UniqueName: \"kubernetes.io/projected/bf676aff-c089-43c6-95de-45c72e2c22ab-kube-api-access-bzxjm\") pod \"bf676aff-c089-43c6-95de-45c72e2c22ab\" (UID: \"bf676aff-c089-43c6-95de-45c72e2c22ab\") " Nov 02 01:29:53.600078 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:53.599957 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf676aff-c089-43c6-95de-45c72e2c22ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf676aff-c089-43c6-95de-45c72e2c22ab-maas-api-openshift-ca\") pod \"bf676aff-c089-43c6-95de-45c72e2c22ab\" (UID: \"bf676aff-c089-43c6-95de-45c72e2c22ab\") " Nov 02 01:29:53.600078 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:53.600011 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf676aff-c089-43c6-95de-45c72e2c22ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf676aff-c089-43c6-95de-45c72e2c22ab-maas-api-xks-ca\") pod \"bf676aff-c089-43c6-95de-45c72e2c22ab\" (UID: \"bf676aff-c089-43c6-95de-45c72e2c22ab\") " Nov 02 01:29:53.600343 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:53.600315 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf676aff-c089-43c6-95de-45c72e2c22ab-maas-api-openshift-ca" pod "bf676aff-c089-43c6-95de-45c72e2c22ab" (UID: "bf676aff-c089-43c6-95de-45c72e2c22ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:29:53.600874 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:53.600756 4427 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf676aff-c089-43c6-95de-45c72e2c22ab-maas-api-openshift-ca\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:29:53.602362 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:53.602339 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf676aff-c089-43c6-95de-45c72e2c22ab-maas-api-xks-ca" pod "bf676aff-c089-43c6-95de-45c72e2c22ab" (UID: "bf676aff-c089-43c6-95de-45c72e2c22ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:29:53.602467 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:53.602394 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf676aff-c089-43c6-95de-45c72e2c22ab-kube-api-access-bzxjm" pod "bf676aff-c089-43c6-95de-45c72e2c22ab" (UID: "bf676aff-c089-43c6-95de-45c72e2c22ab"). InnerVolumeSpecName "kube-api-access-bzxjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:29:53.702522 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:53.702480 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzxjm\" (UniqueName: \"kubernetes.io/projected/bf676aff-c089-43c6-95de-45c72e2c22ab-kube-api-access-bzxjm\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:29:53.702522 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:53.702523 4427 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf676aff-c089-43c6-95de-45c72e2c22ab-maas-api-xks-ca\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:29:54.013213 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.013179 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x"] Nov 02 01:29:54.014101 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.014077 4427 cpu_manager.go:414] "containerMap: removing container" podUID="bf676aff-c089-43c6-95de-45c72e2c22ab" containerName="revoke-keys" Nov 02 01:29:54.014101 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.014097 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf676aff-c089-43c6-95de-45c72e2c22ab" containerName="revoke-keys" Nov 02 01:29:54.014289 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.014189 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf676aff-c089-43c6-95de-45c72e2c22ab" containerName="revoke-keys" Nov 02 01:29:54.018173 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.018132 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x" Nov 02 01:29:54.034248 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.033941 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x"] Nov 02 01:29:54.042467 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.040954 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v"] Nov 02 01:29:54.053774 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.053737 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v" Nov 02 01:29:54.064356 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.063991 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v"] Nov 02 01:29:54.110203 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.110163 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1412019-62bb-4fc3-a7ad-8e123fef90eb-workload-socket\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x\" (UID: \"e1412019-62bb-4fc3-a7ad-8e123fef90eb\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x" Nov 02 01:29:54.110587 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.110563 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dlff\" (UniqueName: \"kubernetes.io/projected/e1412019-62bb-4fc3-a7ad-8e123fef90eb-kube-api-access-2dlff\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x\" (UID: \"e1412019-62bb-4fc3-a7ad-8e123fef90eb\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x" Nov 02 01:29:54.110996 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.110974 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1412019-62bb-4fc3-a7ad-8e123fef90eb-istio-token\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x\" (UID: \"e1412019-62bb-4fc3-a7ad-8e123fef90eb\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x" Nov 02 01:29:54.111231 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.111212 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1412019-62bb-4fc3-a7ad-8e123fef90eb-credential-socket\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x\" (UID: \"e1412019-62bb-4fc3-a7ad-8e123fef90eb\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x" Nov 02 01:29:54.111428 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.111368 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1412019-62bb-4fc3-a7ad-8e123fef90eb-istio-data\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x\" (UID: \"e1412019-62bb-4fc3-a7ad-8e123fef90eb\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x" Nov 02 01:29:54.111948 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.111910 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1412019-62bb-4fc3-a7ad-8e123fef90eb-istio-podinfo\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x\" (UID: \"e1412019-62bb-4fc3-a7ad-8e123fef90eb\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x" Nov 02 01:29:54.112019 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.111992 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1412019-62bb-4fc3-a7ad-8e123fef90eb-workload-certs\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x\" (UID: \"e1412019-62bb-4fc3-a7ad-8e123fef90eb\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x" Nov 02 01:29:54.112062 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.112052 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1412019-62bb-4fc3-a7ad-8e123fef90eb-istio-envoy\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x\" (UID: \"e1412019-62bb-4fc3-a7ad-8e123fef90eb\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x" Nov 02 01:29:54.112281 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.112257 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1412019-62bb-4fc3-a7ad-8e123fef90eb-istiod-ca-cert\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x\" (UID: \"e1412019-62bb-4fc3-a7ad-8e123fef90eb\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x" Nov 02 01:29:54.213937 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.213899 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc786b0e-e9d2-4e73-a48b-42b04f8b3eec-workload-socket\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v\" (UID: \"fc786b0e-e9d2-4e73-a48b-42b04f8b3eec\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v" Nov 02 01:29:54.214120 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.214015 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc786b0e-e9d2-4e73-a48b-42b04f8b3eec-istio-token\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v\" (UID: \"fc786b0e-e9d2-4e73-a48b-42b04f8b3eec\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v" Nov 02 01:29:54.214120 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.214073 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc786b0e-e9d2-4e73-a48b-42b04f8b3eec-credential-socket\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v\" (UID: \"fc786b0e-e9d2-4e73-a48b-42b04f8b3eec\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v" Nov 02 01:29:54.214330 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.214136 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1412019-62bb-4fc3-a7ad-8e123fef90eb-workload-socket\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x\" (UID: \"e1412019-62bb-4fc3-a7ad-8e123fef90eb\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x" Nov 02 01:29:54.214330 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.214191 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dlff\" (UniqueName: \"kubernetes.io/projected/e1412019-62bb-4fc3-a7ad-8e123fef90eb-kube-api-access-2dlff\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x\" (UID: \"e1412019-62bb-4fc3-a7ad-8e123fef90eb\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x" Nov 02 01:29:54.214330 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.214280 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc786b0e-e9d2-4e73-a48b-42b04f8b3eec-istio-podinfo\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v\" (UID: \"fc786b0e-e9d2-4e73-a48b-42b04f8b3eec\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v" Nov 02 01:29:54.214507 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.214352 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc786b0e-e9d2-4e73-a48b-42b04f8b3eec-istio-data\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v\" (UID: \"fc786b0e-e9d2-4e73-a48b-42b04f8b3eec\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v" Nov 02 01:29:54.214507 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.214447 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1412019-62bb-4fc3-a7ad-8e123fef90eb-istio-token\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x\" (UID: \"e1412019-62bb-4fc3-a7ad-8e123fef90eb\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x" Nov 02 01:29:54.214603 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.214507 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc786b0e-e9d2-4e73-a48b-42b04f8b3eec-workload-certs\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v\" (UID: \"fc786b0e-e9d2-4e73-a48b-42b04f8b3eec\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v" Nov 02 01:29:54.214603 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.214546 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1412019-62bb-4fc3-a7ad-8e123fef90eb-credential-socket\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x\" (UID: \"e1412019-62bb-4fc3-a7ad-8e123fef90eb\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x" Nov 02 01:29:54.214603 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.214570 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc786b0e-e9d2-4e73-a48b-42b04f8b3eec-istiod-ca-cert\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v\" (UID: \"fc786b0e-e9d2-4e73-a48b-42b04f8b3eec\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v" Nov 02 01:29:54.214603 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.214600 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1412019-62bb-4fc3-a7ad-8e123fef90eb-istio-data\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x\" (UID: \"e1412019-62bb-4fc3-a7ad-8e123fef90eb\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x" Nov 02 01:29:54.214850 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.214624 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1412019-62bb-4fc3-a7ad-8e123fef90eb-workload-socket\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x\" (UID: \"e1412019-62bb-4fc3-a7ad-8e123fef90eb\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x" Nov 02 01:29:54.214850 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.214649 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1412019-62bb-4fc3-a7ad-8e123fef90eb-istio-podinfo\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x\" (UID: \"e1412019-62bb-4fc3-a7ad-8e123fef90eb\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x" Nov 02 01:29:54.214850 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.214714 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1412019-62bb-4fc3-a7ad-8e123fef90eb-workload-certs\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x\" (UID: \"e1412019-62bb-4fc3-a7ad-8e123fef90eb\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x" Nov 02 01:29:54.214850 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.214781 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1412019-62bb-4fc3-a7ad-8e123fef90eb-istio-envoy\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x\" (UID: \"e1412019-62bb-4fc3-a7ad-8e123fef90eb\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x" Nov 02 01:29:54.215803 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.214976 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1412019-62bb-4fc3-a7ad-8e123fef90eb-credential-socket\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x\" (UID: \"e1412019-62bb-4fc3-a7ad-8e123fef90eb\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x" Nov 02 01:29:54.215803 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.215216 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg747\" (UniqueName: \"kubernetes.io/projected/fc786b0e-e9d2-4e73-a48b-42b04f8b3eec-kube-api-access-xg747\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v\" (UID: \"fc786b0e-e9d2-4e73-a48b-42b04f8b3eec\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v" Nov 02 01:29:54.216945 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.216920 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1412019-62bb-4fc3-a7ad-8e123fef90eb-istiod-ca-cert\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x\" (UID: \"e1412019-62bb-4fc3-a7ad-8e123fef90eb\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x" Nov 02 01:29:54.217117 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.217101 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc786b0e-e9d2-4e73-a48b-42b04f8b3eec-istio-envoy\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v\" (UID: \"fc786b0e-e9d2-4e73-a48b-42b04f8b3eec\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v" Nov 02 01:29:54.217547 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.216943 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1412019-62bb-4fc3-a7ad-8e123fef90eb-workload-certs\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x\" (UID: \"e1412019-62bb-4fc3-a7ad-8e123fef90eb\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x" Nov 02 01:29:54.217893 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.217866 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1412019-62bb-4fc3-a7ad-8e123fef90eb-istio-data\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x\" (UID: \"e1412019-62bb-4fc3-a7ad-8e123fef90eb\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x" Nov 02 01:29:54.218209 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.218183 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1412019-62bb-4fc3-a7ad-8e123fef90eb-istiod-ca-cert\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x\" (UID: \"e1412019-62bb-4fc3-a7ad-8e123fef90eb\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x" Nov 02 01:29:54.220013 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.219990 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1412019-62bb-4fc3-a7ad-8e123fef90eb-istio-envoy\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x\" (UID: \"e1412019-62bb-4fc3-a7ad-8e123fef90eb\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x" Nov 02 01:29:54.220205 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.220186 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1412019-62bb-4fc3-a7ad-8e123fef90eb-istio-podinfo\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x\" (UID: \"e1412019-62bb-4fc3-a7ad-8e123fef90eb\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x" Nov 02 01:29:54.227995 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.227972 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1412019-62bb-4fc3-a7ad-8e123fef90eb-istio-token\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x\" (UID: \"e1412019-62bb-4fc3-a7ad-8e123fef90eb\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x" Nov 02 01:29:54.228290 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.228266 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dlff\" (UniqueName: \"kubernetes.io/projected/e1412019-62bb-4fc3-a7ad-8e123fef90eb-kube-api-access-2dlff\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x\" (UID: \"e1412019-62bb-4fc3-a7ad-8e123fef90eb\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x" Nov 02 01:29:54.264292 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.264213 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj"] Nov 02 01:29:54.281158 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.281123 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj" Nov 02 01:29:54.325129 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.325094 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc786b0e-e9d2-4e73-a48b-42b04f8b3eec-istio-podinfo\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v\" (UID: \"fc786b0e-e9d2-4e73-a48b-42b04f8b3eec\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v" Nov 02 01:29:54.325386 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.325354 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc786b0e-e9d2-4e73-a48b-42b04f8b3eec-istio-data\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v\" (UID: \"fc786b0e-e9d2-4e73-a48b-42b04f8b3eec\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v" Nov 02 01:29:54.325557 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.325542 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc786b0e-e9d2-4e73-a48b-42b04f8b3eec-workload-certs\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v\" (UID: \"fc786b0e-e9d2-4e73-a48b-42b04f8b3eec\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v" Nov 02 01:29:54.325689 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.325674 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc786b0e-e9d2-4e73-a48b-42b04f8b3eec-istiod-ca-cert\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v\" (UID: \"fc786b0e-e9d2-4e73-a48b-42b04f8b3eec\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v" Nov 02 01:29:54.325955 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.325940 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg747\" (UniqueName: \"kubernetes.io/projected/fc786b0e-e9d2-4e73-a48b-42b04f8b3eec-kube-api-access-xg747\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v\" (UID: \"fc786b0e-e9d2-4e73-a48b-42b04f8b3eec\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v" Nov 02 01:29:54.326146 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.326130 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc786b0e-e9d2-4e73-a48b-42b04f8b3eec-istio-envoy\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v\" (UID: \"fc786b0e-e9d2-4e73-a48b-42b04f8b3eec\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v" Nov 02 01:29:54.326256 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.326242 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc786b0e-e9d2-4e73-a48b-42b04f8b3eec-workload-socket\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v\" (UID: \"fc786b0e-e9d2-4e73-a48b-42b04f8b3eec\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v" Nov 02 01:29:54.326359 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.326347 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc786b0e-e9d2-4e73-a48b-42b04f8b3eec-istio-token\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v\" (UID: \"fc786b0e-e9d2-4e73-a48b-42b04f8b3eec\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v" Nov 02 01:29:54.326506 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.326493 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc786b0e-e9d2-4e73-a48b-42b04f8b3eec-credential-socket\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v\" (UID: \"fc786b0e-e9d2-4e73-a48b-42b04f8b3eec\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v" Nov 02 01:29:54.327176 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.327153 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc786b0e-e9d2-4e73-a48b-42b04f8b3eec-credential-socket\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v\" (UID: \"fc786b0e-e9d2-4e73-a48b-42b04f8b3eec\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v" Nov 02 01:29:54.328200 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.327746 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc786b0e-e9d2-4e73-a48b-42b04f8b3eec-istiod-ca-cert\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v\" (UID: \"fc786b0e-e9d2-4e73-a48b-42b04f8b3eec\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v" Nov 02 01:29:54.330632 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.329750 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc786b0e-e9d2-4e73-a48b-42b04f8b3eec-istio-data\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v\" (UID: \"fc786b0e-e9d2-4e73-a48b-42b04f8b3eec\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v" Nov 02 01:29:54.330632 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.330004 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc786b0e-e9d2-4e73-a48b-42b04f8b3eec-workload-certs\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v\" (UID: \"fc786b0e-e9d2-4e73-a48b-42b04f8b3eec\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v" Nov 02 01:29:54.330632 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.330300 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc786b0e-e9d2-4e73-a48b-42b04f8b3eec-workload-socket\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v\" (UID: \"fc786b0e-e9d2-4e73-a48b-42b04f8b3eec\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v" Nov 02 01:29:54.346054 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.337615 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a8ca0d-z4v8m" event={"ID":"bf676aff-c089-43c6-95de-45c72e2c22ab","Type":"ContainerDied","Data":"d7f35d2551774ff13160d1aacdb7866780b3980f3eb8ceee29a6a9ef638c1260"} Nov 02 01:29:54.346054 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.337662 4427 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7f35d2551774ff13160d1aacdb7866780b3980f3eb8ceee29a6a9ef638c1260" Nov 02 01:29:54.352422 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.346301 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03a8ca0d-z4v8m" Nov 02 01:29:54.359503 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.359472 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc786b0e-e9d2-4e73-a48b-42b04f8b3eec-istio-envoy\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v\" (UID: \"fc786b0e-e9d2-4e73-a48b-42b04f8b3eec\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v" Nov 02 01:29:54.359811 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.359791 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj"] Nov 02 01:29:54.361335 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.361312 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x" Nov 02 01:29:54.369042 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.368976 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc786b0e-e9d2-4e73-a48b-42b04f8b3eec-istio-podinfo\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v\" (UID: \"fc786b0e-e9d2-4e73-a48b-42b04f8b3eec\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v" Nov 02 01:29:54.376411 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.376365 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc786b0e-e9d2-4e73-a48b-42b04f8b3eec-istio-token\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v\" (UID: \"fc786b0e-e9d2-4e73-a48b-42b04f8b3eec\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v" Nov 02 01:29:54.377755 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.377731 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg747\" (UniqueName: \"kubernetes.io/projected/fc786b0e-e9d2-4e73-a48b-42b04f8b3eec-kube-api-access-xg747\") pod \"e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v\" (UID: \"fc786b0e-e9d2-4e73-a48b-42b04f8b3eec\") " pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v" Nov 02 01:29:54.384932 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.384906 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v" Nov 02 01:29:54.427979 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.427945 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eec37220-0f11-4a42-847f-363344172167-istiod-ca-cert\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj\" (UID: \"eec37220-0f11-4a42-847f-363344172167\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj" Nov 02 01:29:54.428160 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.428007 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eec37220-0f11-4a42-847f-363344172167-istio-envoy\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj\" (UID: \"eec37220-0f11-4a42-847f-363344172167\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj" Nov 02 01:29:54.428160 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.428027 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eec37220-0f11-4a42-847f-363344172167-istio-podinfo\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj\" (UID: \"eec37220-0f11-4a42-847f-363344172167\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj" Nov 02 01:29:54.428160 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.428086 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eec37220-0f11-4a42-847f-363344172167-workload-socket\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj\" (UID: \"eec37220-0f11-4a42-847f-363344172167\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj" Nov 02 01:29:54.428160 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.428110 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n65v\" (UniqueName: \"kubernetes.io/projected/eec37220-0f11-4a42-847f-363344172167-kube-api-access-7n65v\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj\" (UID: \"eec37220-0f11-4a42-847f-363344172167\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj" Nov 02 01:29:54.428396 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.428184 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eec37220-0f11-4a42-847f-363344172167-istio-token\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj\" (UID: \"eec37220-0f11-4a42-847f-363344172167\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj" Nov 02 01:29:54.428396 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.428213 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eec37220-0f11-4a42-847f-363344172167-workload-certs\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj\" (UID: \"eec37220-0f11-4a42-847f-363344172167\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj" Nov 02 01:29:54.428396 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.428277 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eec37220-0f11-4a42-847f-363344172167-istio-data\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj\" (UID: \"eec37220-0f11-4a42-847f-363344172167\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj" Nov 02 01:29:54.428396 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.428298 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eec37220-0f11-4a42-847f-363344172167-credential-socket\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj\" (UID: \"eec37220-0f11-4a42-847f-363344172167\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj" Nov 02 01:29:54.529952 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.529857 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eec37220-0f11-4a42-847f-363344172167-istio-envoy\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj\" (UID: \"eec37220-0f11-4a42-847f-363344172167\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj" Nov 02 01:29:54.529952 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.529896 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eec37220-0f11-4a42-847f-363344172167-istio-podinfo\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj\" (UID: \"eec37220-0f11-4a42-847f-363344172167\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj" Nov 02 01:29:54.530584 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.529965 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eec37220-0f11-4a42-847f-363344172167-workload-socket\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj\" (UID: \"eec37220-0f11-4a42-847f-363344172167\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj" Nov 02 01:29:54.530584 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.529996 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n65v\" (UniqueName: \"kubernetes.io/projected/eec37220-0f11-4a42-847f-363344172167-kube-api-access-7n65v\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj\" (UID: \"eec37220-0f11-4a42-847f-363344172167\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj" Nov 02 01:29:54.530584 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.530110 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eec37220-0f11-4a42-847f-363344172167-istio-token\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj\" (UID: \"eec37220-0f11-4a42-847f-363344172167\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj" Nov 02 01:29:54.530584 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.530155 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eec37220-0f11-4a42-847f-363344172167-workload-certs\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj\" (UID: \"eec37220-0f11-4a42-847f-363344172167\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj" Nov 02 01:29:54.530584 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.530251 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eec37220-0f11-4a42-847f-363344172167-istio-data\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj\" (UID: \"eec37220-0f11-4a42-847f-363344172167\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj" Nov 02 01:29:54.530584 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.530289 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eec37220-0f11-4a42-847f-363344172167-credential-socket\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj\" (UID: \"eec37220-0f11-4a42-847f-363344172167\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj" Nov 02 01:29:54.530584 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.530407 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eec37220-0f11-4a42-847f-363344172167-istiod-ca-cert\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj\" (UID: \"eec37220-0f11-4a42-847f-363344172167\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj" Nov 02 01:29:54.530949 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.530841 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eec37220-0f11-4a42-847f-363344172167-workload-socket\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj\" (UID: \"eec37220-0f11-4a42-847f-363344172167\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj" Nov 02 01:29:54.531700 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.531634 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eec37220-0f11-4a42-847f-363344172167-workload-certs\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj\" (UID: \"eec37220-0f11-4a42-847f-363344172167\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj" Nov 02 01:29:54.531700 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.531638 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eec37220-0f11-4a42-847f-363344172167-istiod-ca-cert\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj\" (UID: \"eec37220-0f11-4a42-847f-363344172167\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj" Nov 02 01:29:54.532156 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.532112 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eec37220-0f11-4a42-847f-363344172167-istio-data\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj\" (UID: \"eec37220-0f11-4a42-847f-363344172167\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj" Nov 02 01:29:54.532774 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.532352 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eec37220-0f11-4a42-847f-363344172167-credential-socket\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj\" (UID: \"eec37220-0f11-4a42-847f-363344172167\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj" Nov 02 01:29:54.533071 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.533048 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eec37220-0f11-4a42-847f-363344172167-istio-envoy\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj\" (UID: \"eec37220-0f11-4a42-847f-363344172167\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj" Nov 02 01:29:54.534485 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.534461 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eec37220-0f11-4a42-847f-363344172167-istio-podinfo\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj\" (UID: \"eec37220-0f11-4a42-847f-363344172167\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj" Nov 02 01:29:54.544107 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.544056 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eec37220-0f11-4a42-847f-363344172167-istio-token\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj\" (UID: \"eec37220-0f11-4a42-847f-363344172167\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj" Nov 02 01:29:54.544805 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.544763 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n65v\" (UniqueName: \"kubernetes.io/projected/eec37220-0f11-4a42-847f-363344172167-kube-api-access-7n65v\") pod \"e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj\" (UID: \"eec37220-0f11-4a42-847f-363344172167\") " pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj" Nov 02 01:29:54.633273 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.633241 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj" Nov 02 01:29:54.675298 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:29:54.675265 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1412019_62bb_4fc3_a7ad_8e123fef90eb.slice/crio-29708cb6ab707868d112cb40dae2225967bd7d66e70c6a04d5e048e42c366e9e WatchSource:0}: Error finding container 29708cb6ab707868d112cb40dae2225967bd7d66e70c6a04d5e048e42c366e9e: Status 404 returned error can't find the container with id 29708cb6ab707868d112cb40dae2225967bd7d66e70c6a04d5e048e42c366e9e Nov 02 01:29:54.678173 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.678138 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:29:54.678271 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.678223 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:29:54.678328 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.678270 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:29:54.679775 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.679748 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x"] Nov 02 01:29:54.713236 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.704509 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v"] Nov 02 01:29:54.725572 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.717577 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:29:54.725572 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.717657 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:29:54.725572 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.717700 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:29:54.929575 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.929418 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj"] Nov 02 01:29:54.930789 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:29:54.930763 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeec37220_0f11_4a42_847f_363344172167.slice/crio-1de3d83b70990084b27410b5075685ad6b6b9217f81497ef9037c4052a27a7ed WatchSource:0}: Error finding container 1de3d83b70990084b27410b5075685ad6b6b9217f81497ef9037c4052a27a7ed: Status 404 returned error can't find the container with id 1de3d83b70990084b27410b5075685ad6b6b9217f81497ef9037c4052a27a7ed Nov 02 01:29:54.933610 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.933578 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:29:54.933673 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.933643 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:29:54.933673 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:54.933670 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:29:55.421838 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:55.421786 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x" event={"ID":"e1412019-62bb-4fc3-a7ad-8e123fef90eb","Type":"ContainerStarted","Data":"66f3e69d92a5ab91170fb508ebaa7f30900ee72f6a5a4e3fce4255db6ddb5fc6"} Nov 02 01:29:55.422084 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:55.421898 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x" event={"ID":"e1412019-62bb-4fc3-a7ad-8e123fef90eb","Type":"ContainerStarted","Data":"29708cb6ab707868d112cb40dae2225967bd7d66e70c6a04d5e048e42c366e9e"} Nov 02 01:29:55.443464 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:55.429480 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v" event={"ID":"fc786b0e-e9d2-4e73-a48b-42b04f8b3eec","Type":"ContainerStarted","Data":"d9a44c2c3ab06359b52cb7b257e29f013b76aab16906d76e84a7b55e8b0898c5"} Nov 02 01:29:55.443464 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:55.429548 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v" event={"ID":"fc786b0e-e9d2-4e73-a48b-42b04f8b3eec","Type":"ContainerStarted","Data":"56ec173597fcd3971e892034c48f37046713b4c9307ac0f56c0482333fd92353"} Nov 02 01:29:55.443464 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:55.440280 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj" event={"ID":"eec37220-0f11-4a42-847f-363344172167","Type":"ContainerStarted","Data":"969856304df23df7a3ca8ac178919007726c6bb6c89de29257e3ea52d027456f"} Nov 02 01:29:55.443464 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:55.440345 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj" event={"ID":"eec37220-0f11-4a42-847f-363344172167","Type":"ContainerStarted","Data":"1de3d83b70990084b27410b5075685ad6b6b9217f81497ef9037c4052a27a7ed"} Nov 02 01:29:55.448219 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:55.448156 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x" podStartSLOduration=2.448138276 podStartE2EDuration="2.448138276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:29:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:29:55.447445825 +0000 UTC m=+900.068451037" watchObservedRunningTime="2026-11-02 01:29:55.448138276 +0000 UTC m=+900.069143487" Nov 02 01:29:55.494520 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:55.494455 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj" podStartSLOduration=1.494433587 podStartE2EDuration="1.494433587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:29:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:29:55.490859334 +0000 UTC m=+900.111864558" watchObservedRunningTime="2026-11-02 01:29:55.494433587 +0000 UTC m=+900.115438802" Nov 02 01:29:55.519429 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:55.519338 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v" podStartSLOduration=1.519315483 podStartE2EDuration="1.519315483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:29:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:29:55.512908946 +0000 UTC m=+900.133914157" watchObservedRunningTime="2026-11-02 01:29:55.519315483 +0000 UTC m=+900.140320749" Nov 02 01:29:55.633873 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:55.633838 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj" Nov 02 01:29:56.376403 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:56.368704 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x" Nov 02 01:29:56.376403 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:56.368833 4427 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 01:29:56.376403 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:56.368876 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x" podUID="e1412019-62bb-4fc3-a7ad-8e123fef90eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 01:29:56.402400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:56.399513 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v" Nov 02 01:29:56.402400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:56.401655 4427 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 01:29:56.402400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:56.401702 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v" podUID="fc786b0e-e9d2-4e73-a48b-42b04f8b3eec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 01:29:56.635676 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:56.635582 4427 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 02 01:29:56.636198 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:56.636168 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj" podUID="eec37220-0f11-4a42-847f-363344172167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 02 01:29:57.280527 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:57.280467 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" podUID="366c78ee-4aaf-4b7c-84b4-17028538cb1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:57.281133 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:57.281103 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" podUID="eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Nov 02 01:29:57.341407 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:57.340884 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" podUID="c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Nov 02 01:29:57.345396 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:57.342015 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" podUID="d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:57.345396 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:57.342310 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" podUID="7085c0ab-7ee3-4665-abad-2977c25f68ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:57.396403 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:57.383834 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" podUID="353a8a41-21ee-4470-861c-b53833beb106" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 02 01:29:57.396403 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:57.389776 4427 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 01:29:57.396403 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:57.389878 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x" podUID="e1412019-62bb-4fc3-a7ad-8e123fef90eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 01:29:57.401825 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:57.401142 4427 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 01:29:57.401825 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:57.401728 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v" podUID="fc786b0e-e9d2-4e73-a48b-42b04f8b3eec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 01:29:57.408396 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:57.402422 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" podUID="ea61971c-5a72-4609-b5a0-58a74b324bb7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:57.412397 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:57.408777 4427 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" podUID="cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": context deadline exceeded" Nov 02 01:29:57.649890 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:57.649849 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" podUID="fb533b5f-3525-454a-8ce2-ea9246194f28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Nov 02 01:29:57.676428 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:57.660615 4427 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 02 01:29:57.676428 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:57.660671 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj" podUID="eec37220-0f11-4a42-847f-363344172167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 02 01:29:57.713312 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:57.712476 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" podUID="cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:57.791398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:57.787720 4427 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" podUID="9c056798-2cf8-47b1-9829-67cebda7a248" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:57.949317 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:57.948526 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" podUID="706df1bd-19cf-4ab9-9aaa-3c12c97ccf45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 02 01:29:57.974479 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:57.973450 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" podUID="47a9069f-1ffa-4c83-be8b-5e62f1d3275a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 02 01:29:58.042531 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:58.042480 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" podUID="d73ae484-a9b8-4565-a346-0d803a1981ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:58.103512 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:58.103466 4427 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" podUID="fb533b5f-3525-454a-8ce2-ea9246194f28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:58.117507 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:58.117459 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" podUID="9c056798-2cf8-47b1-9829-67cebda7a248" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:58.238587 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:58.238464 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" podUID="f67c954a-ce5b-41d9-8d99-7d126bd89d59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:58.292316 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:58.292267 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" podUID="eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:58.293423 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:58.292450 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" podUID="366c78ee-4aaf-4b7c-84b4-17028538cb1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 02 01:29:58.294857 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:58.294007 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" Nov 02 01:29:58.295462 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:58.295448 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" Nov 02 01:29:58.355767 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:58.352484 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" podUID="7085c0ab-7ee3-4665-abad-2977c25f68ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:58.355767 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:58.352881 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" podUID="c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Nov 02 01:29:58.355767 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:58.353158 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" podUID="d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Nov 02 01:29:58.355767 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:58.353780 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" Nov 02 01:29:58.355767 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:58.353834 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" Nov 02 01:29:58.355767 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:58.353866 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" Nov 02 01:29:58.364037 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:58.363913 4427 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 01:29:58.364037 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:58.363980 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x" podUID="e1412019-62bb-4fc3-a7ad-8e123fef90eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 01:29:58.387131 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:58.384996 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" podUID="353a8a41-21ee-4470-861c-b53833beb106" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:58.387131 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:58.385416 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" Nov 02 01:29:58.387131 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:58.387014 4427 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 01:29:58.387131 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:58.387055 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v" podUID="fc786b0e-e9d2-4e73-a48b-42b04f8b3eec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 01:29:58.411028 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:58.410942 4427 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" podUID="c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:58.645462 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:58.641136 4427 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 02 01:29:58.645462 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:58.641189 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj" podUID="eec37220-0f11-4a42-847f-363344172167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 02 01:29:58.659414 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:58.653619 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" podUID="fb533b5f-3525-454a-8ce2-ea9246194f28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Nov 02 01:29:58.682509 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:58.682464 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" Nov 02 01:29:58.746397 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:58.729211 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" podUID="cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:58.869692 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:58.869649 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" Nov 02 01:29:58.949340 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:58.949142 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" podUID="706df1bd-19cf-4ab9-9aaa-3c12c97ccf45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:59.003761 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:59.003710 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" podUID="47a9069f-1ffa-4c83-be8b-5e62f1d3275a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 02 01:29:59.036865 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:59.035537 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" Nov 02 01:29:59.037081 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:59.036918 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" Nov 02 01:29:59.067809 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:59.067757 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" podUID="d73ae484-a9b8-4565-a346-0d803a1981ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": context deadline exceeded" Nov 02 01:29:59.069405 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:59.069358 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" Nov 02 01:29:59.124431 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:59.124281 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" podUID="9c056798-2cf8-47b1-9829-67cebda7a248" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 01:29:59.195811 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:59.194942 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" Nov 02 01:29:59.254541 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:59.254415 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" podUID="f67c954a-ce5b-41d9-8d99-7d126bd89d59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:59.266462 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:59.257858 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" Nov 02 01:29:59.295114 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:59.295049 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" podUID="eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:59.295610 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:59.295535 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" podUID="366c78ee-4aaf-4b7c-84b4-17028538cb1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:59.366213 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:59.366157 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" podUID="d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:59.370685 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:59.366512 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" podUID="7085c0ab-7ee3-4665-abad-2977c25f68ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Nov 02 01:29:59.463582 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:59.365793 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" podUID="c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Nov 02 01:29:59.464063 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:59.396179 4427 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 01:29:59.464244 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:59.464219 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v" podUID="fc786b0e-e9d2-4e73-a48b-42b04f8b3eec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 01:29:59.465153 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:59.465120 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" podUID="353a8a41-21ee-4470-861c-b53833beb106" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 02 01:29:59.465582 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:59.465560 4427 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 01:29:59.465744 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:59.465722 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x" podUID="e1412019-62bb-4fc3-a7ad-8e123fef90eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 01:29:59.476025 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:59.475985 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" podUID="ea61971c-5a72-4609-b5a0-58a74b324bb7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:29:59.651407 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:59.647544 4427 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 02 01:29:59.687570 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:59.687522 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj" podUID="eec37220-0f11-4a42-847f-363344172167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 02 01:29:59.699473 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:59.695955 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" podUID="fb533b5f-3525-454a-8ce2-ea9246194f28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Nov 02 01:30:00.178460 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:59.948502 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" Nov 02 01:30:00.178460 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:59.961334 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" podUID="706df1bd-19cf-4ab9-9aaa-3c12c97ccf45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:00.178460 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:29:59.974525 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" podUID="cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:00.178460 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:00.038518 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" podUID="47a9069f-1ffa-4c83-be8b-5e62f1d3275a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 02 01:30:00.178460 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:00.084428 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" podUID="d73ae484-a9b8-4565-a346-0d803a1981ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:00.204451 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:00.195494 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" podUID="9c056798-2cf8-47b1-9829-67cebda7a248" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:00.263062 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:00.263017 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" podUID="f67c954a-ce5b-41d9-8d99-7d126bd89d59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:00.330079 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:00.315928 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" podUID="eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:00.350607 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:00.342460 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" podUID="366c78ee-4aaf-4b7c-84b4-17028538cb1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:00.386265 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:00.386230 4427 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 01:30:00.386535 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:00.386507 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x" podUID="e1412019-62bb-4fc3-a7ad-8e123fef90eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 01:30:00.397494 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:00.388143 4427 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 01:30:00.445892 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:00.445790 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v" podUID="fc786b0e-e9d2-4e73-a48b-42b04f8b3eec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 01:30:00.529539 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:00.529488 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" podUID="ea61971c-5a72-4609-b5a0-58a74b324bb7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:00.621683 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:00.530466 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" podUID="d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:00.640873 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:00.603676 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" podUID="7085c0ab-7ee3-4665-abad-2977c25f68ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Nov 02 01:30:00.640873 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:00.603715 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" podUID="c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:00.677974 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:00.623087 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" Nov 02 01:30:00.835955 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:00.835842 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" podUID="fb533b5f-3525-454a-8ce2-ea9246194f28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Nov 02 01:30:01.021243 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:00.875291 4427 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 02 01:30:01.048530 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:01.048482 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj" podUID="eec37220-0f11-4a42-847f-363344172167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 02 01:30:01.048796 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:01.042733 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" podUID="47a9069f-1ffa-4c83-be8b-5e62f1d3275a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:01.049494 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:01.023511 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" podUID="706df1bd-19cf-4ab9-9aaa-3c12c97ccf45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 02 01:30:01.074505 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:01.043143 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" podUID="cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:01.117465 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:01.109089 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" podUID="d73ae484-a9b8-4565-a346-0d803a1981ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:01.238084 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:01.198311 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" podUID="9c056798-2cf8-47b1-9829-67cebda7a248" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:01.813402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:01.283475 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" podUID="f67c954a-ce5b-41d9-8d99-7d126bd89d59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 02 01:30:01.813402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:01.395849 4427 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 01:30:01.813402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:01.753318 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x" podUID="e1412019-62bb-4fc3-a7ad-8e123fef90eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 01:30:01.813402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:01.397133 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" podUID="eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:01.830403 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:01.397073 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" podUID="366c78ee-4aaf-4b7c-84b4-17028538cb1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:01.842410 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:01.397023 4427 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 01:30:01.855538 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:01.855492 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v" podUID="fc786b0e-e9d2-4e73-a48b-42b04f8b3eec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 01:30:01.859452 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:01.660714 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-123986-29893050-szk9t"] Nov 02 01:30:01.966889 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:01.940871 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" podUID="fb533b5f-3525-454a-8ce2-ea9246194f28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:01.968542 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:01.840980 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" podUID="7085c0ab-7ee3-4665-abad-2977c25f68ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Nov 02 01:30:01.980485 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:01.634970 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" podUID="d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Nov 02 01:30:01.982291 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:01.635674 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" podUID="ea61971c-5a72-4609-b5a0-58a74b324bb7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 02 01:30:01.991759 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:01.841036 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" podUID="c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:02.007010 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:01.894745 4427 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 02 01:30:02.007395 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:02.007310 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj" podUID="eec37220-0f11-4a42-847f-363344172167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 02 01:30:02.008758 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:01.661242 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" podUID="353a8a41-21ee-4470-861c-b53833beb106" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 02 01:30:02.030527 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:01.904101 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-nddkv" podUID="4379e4b2-e68b-4ddf-a3bf-9ec404e61f56" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.18:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:02.146765 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:30:02.146725 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:30:02.248405 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:02.233744 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-123986-29893050-szk9t" Nov 02 01:30:02.398153 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:02.259660 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" podUID="47a9069f-1ffa-4c83-be8b-5e62f1d3275a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 02 01:30:02.399087 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:02.259695 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" podUID="706df1bd-19cf-4ab9-9aaa-3c12c97ccf45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 02 01:30:02.399406 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:02.259727 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" podUID="cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 02 01:30:02.403612 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:02.259788 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" podUID="d73ae484-a9b8-4565-a346-0d803a1981ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": context deadline exceeded" Nov 02 01:30:02.407398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:02.359351 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k6h7\" (UniqueName: \"kubernetes.io/projected/9423ce2b-b5fb-4a3b-ae5a-d33802a0a2f8-kube-api-access-4k6h7\") pod \"maas-api-key-cleanup-e2e-worker-w5-123986-29893050-szk9t\" (UID: \"9423ce2b-b5fb-4a3b-ae5a-d33802a0a2f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-123986-29893050-szk9t" Nov 02 01:30:02.432980 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:02.367868 4427 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" podUID="d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:02.432980 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:02.395543 4427 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 01:30:02.432980 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:02.432704 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x" podUID="e1412019-62bb-4fc3-a7ad-8e123fef90eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 01:30:02.432980 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:02.396780 4427 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 01:30:02.432980 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:02.432827 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v" podUID="fc786b0e-e9d2-4e73-a48b-42b04f8b3eec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 01:30:02.434081 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:02.434007 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9423ce2b-b5fb-4a3b-ae5a-d33802a0a2f8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-123986-29893050-szk9t\" (UID: \"9423ce2b-b5fb-4a3b-ae5a-d33802a0a2f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-123986-29893050-szk9t" Nov 02 01:30:02.681478 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:02.680615 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893050-pbqfk"] Nov 02 01:30:02.768406 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:02.760469 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" podUID="9c056798-2cf8-47b1-9829-67cebda7a248" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 01:30:02.787525 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:02.768723 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" podUID="f67c954a-ce5b-41d9-8d99-7d126bd89d59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 02 01:30:02.787525 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:02.777474 4427 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 01:30:02.787525 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:02.777535 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8" podUID="af735a61-695b-4a66-a7cb-d0df06b3b58b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:02.787525 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:02.777739 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" podUID="eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Nov 02 01:30:02.817454 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:02.788936 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\"" Nov 02 01:30:02.821425 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:02.821286 4427 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 02 01:30:02.821425 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:02.821346 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj" podUID="eec37220-0f11-4a42-847f-363344172167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 02 01:30:02.839412 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:02.828972 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9423ce2b-b5fb-4a3b-ae5a-d33802a0a2f8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-123986-29893050-szk9t\" (UID: \"9423ce2b-b5fb-4a3b-ae5a-d33802a0a2f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-123986-29893050-szk9t" Nov 02 01:30:02.839835 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:02.838514 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" podUID="366c78ee-4aaf-4b7c-84b4-17028538cb1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:02.839980 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:02.839953 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k6h7\" (UniqueName: \"kubernetes.io/projected/9423ce2b-b5fb-4a3b-ae5a-d33802a0a2f8-kube-api-access-4k6h7\") pod \"maas-api-key-cleanup-e2e-worker-w5-123986-29893050-szk9t\" (UID: \"9423ce2b-b5fb-4a3b-ae5a-d33802a0a2f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-123986-29893050-szk9t" Nov 02 01:30:02.884549 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:02.839422 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9423ce2b-b5fb-4a3b-ae5a-d33802a0a2f8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-123986-29893050-szk9t\" (UID: \"9423ce2b-b5fb-4a3b-ae5a-d33802a0a2f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-123986-29893050-szk9t" Nov 02 01:30:02.903800 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:02.867504 4427 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" podUID="d73ae484-a9b8-4565-a346-0d803a1981ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": context deadline exceeded" Nov 02 01:30:02.990464 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:02.974856 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" podUID="fb533b5f-3525-454a-8ce2-ea9246194f28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:03.035402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:03.031551 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" podUID="c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:03.055508 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:03.055465 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" podUID="7085c0ab-7ee3-4665-abad-2977c25f68ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Nov 02 01:30:03.056577 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:03.056137 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" podUID="ea61971c-5a72-4609-b5a0-58a74b324bb7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 02 01:30:03.057473 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:03.056167 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" podUID="d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Nov 02 01:30:03.057996 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:03.056191 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" podUID="353a8a41-21ee-4470-861c-b53833beb106" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 02 01:30:03.140155 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:03.140121 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k6h7\" (UniqueName: \"kubernetes.io/projected/9423ce2b-b5fb-4a3b-ae5a-d33802a0a2f8-kube-api-access-4k6h7\") pod \"maas-api-key-cleanup-e2e-worker-w5-123986-29893050-szk9t\" (UID: \"9423ce2b-b5fb-4a3b-ae5a-d33802a0a2f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-123986-29893050-szk9t" Nov 02 01:30:03.167717 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:03.167678 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b6ec07-29893050-x5z5c"] Nov 02 01:30:03.205496 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:03.168027 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893050-pbqfk" Nov 02 01:30:03.278328 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:03.266649 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" Nov 02 01:30:03.278328 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:03.266686 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-123986-29893050-szk9t"] Nov 02 01:30:03.278328 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:03.266721 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893050-pbqfk"] Nov 02 01:30:03.278328 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:03.266750 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5687bc-29893050-bvwhp"] Nov 02 01:30:03.278328 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:03.267324 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b6ec07-29893050-x5z5c" Nov 02 01:30:03.292109 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:03.292077 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-123986-29893050-szk9t" Nov 02 01:30:03.321399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:03.320586 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b6ec07-29893050-x5z5c"] Nov 02 01:30:03.321399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:03.320727 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5687bc-29893050-bvwhp" Nov 02 01:30:03.362535 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:03.362505 4427 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 01:30:03.373401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:03.362743 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x" podUID="e1412019-62bb-4fc3-a7ad-8e123fef90eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 01:30:03.404404 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:03.371605 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5687bc-29893050-bvwhp"] Nov 02 01:30:03.587014 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:03.423916 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76a550cd-124f-45c4-9c6a-77f1eb8f0c97-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b6ec07-29893050-x5z5c\" (UID: \"76a550cd-124f-45c4-9c6a-77f1eb8f0c97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b6ec07-29893050-x5z5c" Nov 02 01:30:03.587326 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:03.587302 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xkhj\" (UniqueName: \"kubernetes.io/projected/76a550cd-124f-45c4-9c6a-77f1eb8f0c97-kube-api-access-5xkhj\") pod \"maas-api-key-cleanup-e2e-worker-w4-b6ec07-29893050-x5z5c\" (UID: \"76a550cd-124f-45c4-9c6a-77f1eb8f0c97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b6ec07-29893050-x5z5c" Nov 02 01:30:03.587619 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:03.587598 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtfcc\" (UniqueName: \"kubernetes.io/projected/0c82e794-a8fd-4667-8433-c1879a6955b8-kube-api-access-xtfcc\") pod \"maas-api-key-cleanup-29893050-pbqfk\" (UID: \"0c82e794-a8fd-4667-8433-c1879a6955b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893050-pbqfk" Nov 02 01:30:03.587857 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:03.587841 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c82e794-a8fd-4667-8433-c1879a6955b8-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893050-pbqfk\" (UID: \"0c82e794-a8fd-4667-8433-c1879a6955b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893050-pbqfk" Nov 02 01:30:03.594734 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:03.424026 4427 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 01:30:03.595011 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:03.594965 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v" podUID="fc786b0e-e9d2-4e73-a48b-42b04f8b3eec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 01:30:03.595669 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:03.425486 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" podUID="d73ae484-a9b8-4565-a346-0d803a1981ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:03.596671 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:03.437965 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" podUID="47a9069f-1ffa-4c83-be8b-5e62f1d3275a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:03.612844 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:03.425530 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" podUID="706df1bd-19cf-4ab9-9aaa-3c12c97ccf45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:03.613170 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:03.426247 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" podUID="cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:03.613837 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:03.443607 4427 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 01:30:03.613955 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:03.613843 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2" podUID="2bcaee75-ac15-4b13-8f60-401ee60464ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:03.652802 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:03.638973 4427 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 02 01:30:03.652802 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:03.639034 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj" podUID="eec37220-0f11-4a42-847f-363344172167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 02 01:30:03.737863 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:03.733866 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76a550cd-124f-45c4-9c6a-77f1eb8f0c97-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b6ec07-29893050-x5z5c\" (UID: \"76a550cd-124f-45c4-9c6a-77f1eb8f0c97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b6ec07-29893050-x5z5c" Nov 02 01:30:03.737863 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:03.733953 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xkhj\" (UniqueName: \"kubernetes.io/projected/76a550cd-124f-45c4-9c6a-77f1eb8f0c97-kube-api-access-5xkhj\") pod \"maas-api-key-cleanup-e2e-worker-w4-b6ec07-29893050-x5z5c\" (UID: \"76a550cd-124f-45c4-9c6a-77f1eb8f0c97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b6ec07-29893050-x5z5c" Nov 02 01:30:03.737863 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:03.734017 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtfcc\" (UniqueName: \"kubernetes.io/projected/0c82e794-a8fd-4667-8433-c1879a6955b8-kube-api-access-xtfcc\") pod \"maas-api-key-cleanup-29893050-pbqfk\" (UID: \"0c82e794-a8fd-4667-8433-c1879a6955b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893050-pbqfk" Nov 02 01:30:03.737863 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:03.734092 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c82e794-a8fd-4667-8433-c1879a6955b8-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893050-pbqfk\" (UID: \"0c82e794-a8fd-4667-8433-c1879a6955b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893050-pbqfk" Nov 02 01:30:03.737863 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:03.734171 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04f12dc1-5c1e-42ea-a49b-7bc46cf59568-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5687bc-29893050-bvwhp\" (UID: \"04f12dc1-5c1e-42ea-a49b-7bc46cf59568\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5687bc-29893050-bvwhp" Nov 02 01:30:03.737863 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:03.734214 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgbmr\" (UniqueName: \"kubernetes.io/projected/04f12dc1-5c1e-42ea-a49b-7bc46cf59568-kube-api-access-sgbmr\") pod \"maas-api-key-cleanup-e2e-worker-w1-5687bc-29893050-bvwhp\" (UID: \"04f12dc1-5c1e-42ea-a49b-7bc46cf59568\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5687bc-29893050-bvwhp" Nov 02 01:30:03.737863 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:03.735250 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76a550cd-124f-45c4-9c6a-77f1eb8f0c97-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b6ec07-29893050-x5z5c\" (UID: \"76a550cd-124f-45c4-9c6a-77f1eb8f0c97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b6ec07-29893050-x5z5c" Nov 02 01:30:03.774277 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:03.766162 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" podUID="9c056798-2cf8-47b1-9829-67cebda7a248" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:03.775045 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:03.774642 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" podUID="f67c954a-ce5b-41d9-8d99-7d126bd89d59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:03.818474 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:03.806490 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" podUID="eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:03.833675 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:03.833640 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xkhj\" (UniqueName: \"kubernetes.io/projected/76a550cd-124f-45c4-9c6a-77f1eb8f0c97-kube-api-access-5xkhj\") pod \"maas-api-key-cleanup-e2e-worker-w4-b6ec07-29893050-x5z5c\" (UID: \"76a550cd-124f-45c4-9c6a-77f1eb8f0c97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b6ec07-29893050-x5z5c" Nov 02 01:30:03.903537 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:03.842229 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" podUID="366c78ee-4aaf-4b7c-84b4-17028538cb1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:03.903537 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:03.849999 4427 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" podUID="ea61971c-5a72-4609-b5a0-58a74b324bb7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:03.903537 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:03.850590 4427 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 01:30:03.903537 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:03.850633 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28" podUID="1563efef-2ed3-4347-8399-4c9ca520db4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:03.903537 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:03.851811 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04f12dc1-5c1e-42ea-a49b-7bc46cf59568-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5687bc-29893050-bvwhp\" (UID: \"04f12dc1-5c1e-42ea-a49b-7bc46cf59568\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5687bc-29893050-bvwhp" Nov 02 01:30:03.903537 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:03.851868 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgbmr\" (UniqueName: \"kubernetes.io/projected/04f12dc1-5c1e-42ea-a49b-7bc46cf59568-kube-api-access-sgbmr\") pod \"maas-api-key-cleanup-e2e-worker-w1-5687bc-29893050-bvwhp\" (UID: \"04f12dc1-5c1e-42ea-a49b-7bc46cf59568\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5687bc-29893050-bvwhp" Nov 02 01:30:03.903537 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:03.877789 4427 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 01:30:03.903537 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:03.902673 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4" podUID="eb3258a9-adfb-46de-a0ab-7b4649948689" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:03.950091 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:03.925362 4427 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" podUID="706df1bd-19cf-4ab9-9aaa-3c12c97ccf45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:03.964293 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:03.926151 4427 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 01:30:03.964293 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:03.962631 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z" podUID="006dbe2b-932c-4c07-9430-51b9bd3b2bb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:03.964293 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:03.941361 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgbmr\" (UniqueName: \"kubernetes.io/projected/04f12dc1-5c1e-42ea-a49b-7bc46cf59568-kube-api-access-sgbmr\") pod \"maas-api-key-cleanup-e2e-worker-w1-5687bc-29893050-bvwhp\" (UID: \"04f12dc1-5c1e-42ea-a49b-7bc46cf59568\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5687bc-29893050-bvwhp" Nov 02 01:30:03.986338 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:03.985941 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" podUID="fb533b5f-3525-454a-8ce2-ea9246194f28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:03.986789 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:03.986756 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04f12dc1-5c1e-42ea-a49b-7bc46cf59568-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5687bc-29893050-bvwhp\" (UID: \"04f12dc1-5c1e-42ea-a49b-7bc46cf59568\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5687bc-29893050-bvwhp" Nov 02 01:30:04.010035 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:04.010000 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b6ec07-29893050-x5z5c" Nov 02 01:30:04.010759 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:04.010731 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c82e794-a8fd-4667-8433-c1879a6955b8-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893050-pbqfk\" (UID: \"0c82e794-a8fd-4667-8433-c1879a6955b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893050-pbqfk" Nov 02 01:30:04.034628 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:04.034583 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" podUID="c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:04.036879 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:04.036841 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5687bc-29893050-bvwhp" Nov 02 01:30:04.066648 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:04.066429 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" podUID="7085c0ab-7ee3-4665-abad-2977c25f68ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:04.079644 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:04.079541 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" podUID="353a8a41-21ee-4470-861c-b53833beb106" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:04.088406 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:04.079870 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" podUID="ea61971c-5a72-4609-b5a0-58a74b324bb7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:04.097572 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:04.080000 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" podUID="d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:04.098973 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:04.079847 4427 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" podUID="47a9069f-1ffa-4c83-be8b-5e62f1d3275a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": context deadline exceeded" Nov 02 01:30:04.310780 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:04.310740 4427 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 01:30:04.311059 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:04.311031 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2" podUID="e1a145f0-821a-4d90-a373-fd65c74c8e57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:04.370271 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:04.362317 4427 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 01:30:04.370271 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:04.362390 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x" podUID="e1412019-62bb-4fc3-a7ad-8e123fef90eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 01:30:04.370271 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:04.370149 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtfcc\" (UniqueName: \"kubernetes.io/projected/0c82e794-a8fd-4667-8433-c1879a6955b8-kube-api-access-xtfcc\") pod \"maas-api-key-cleanup-29893050-pbqfk\" (UID: \"0c82e794-a8fd-4667-8433-c1879a6955b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893050-pbqfk" Nov 02 01:30:04.408964 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:04.389995 4427 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 01:30:04.408964 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:04.390052 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v" podUID="fc786b0e-e9d2-4e73-a48b-42b04f8b3eec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 01:30:04.506404 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:04.504475 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893050-pbqfk" Nov 02 01:30:04.635861 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:04.601501 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" podUID="47a9069f-1ffa-4c83-be8b-5e62f1d3275a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:04.635861 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:04.614634 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" podUID="cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:04.635861 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:04.614973 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" podUID="706df1bd-19cf-4ab9-9aaa-3c12c97ccf45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 02 01:30:04.635861 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:04.627533 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" podUID="d73ae484-a9b8-4565-a346-0d803a1981ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": context deadline exceeded" Nov 02 01:30:04.647146 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:04.647106 4427 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 02 01:30:04.647434 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:04.647404 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj" podUID="eec37220-0f11-4a42-847f-363344172167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 02 01:30:04.886462 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:04.837496 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" podUID="f67c954a-ce5b-41d9-8d99-7d126bd89d59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:04.886462 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:04.837939 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" podUID="9c056798-2cf8-47b1-9829-67cebda7a248" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 01:30:04.903037 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:04.886971 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" podUID="366c78ee-4aaf-4b7c-84b4-17028538cb1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:04.903037 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:04.840474 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" podUID="eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:05.008596 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:05.008520 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" podUID="fb533b5f-3525-454a-8ce2-ea9246194f28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:05.206516 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:05.045419 4427 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 01:30:05.206516 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:05.045485 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv" podUID="d5cb7660-e2aa-48a2-9eac-508cd83f93cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:05.206516 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:05.054978 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" podUID="c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:05.206516 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:05.112490 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" podUID="d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:05.206516 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:05.112917 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" podUID="7085c0ab-7ee3-4665-abad-2977c25f68ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Nov 02 01:30:05.206516 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:05.113213 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" podUID="ea61971c-5a72-4609-b5a0-58a74b324bb7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:05.444318 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:05.319852 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" podUID="353a8a41-21ee-4470-861c-b53833beb106" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 02 01:30:05.444318 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:05.361992 4427 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 01:30:05.444318 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:05.362054 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x" podUID="e1412019-62bb-4fc3-a7ad-8e123fef90eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 01:30:05.444318 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:05.362339 4427 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" podUID="7085c0ab-7ee3-4665-abad-2977c25f68ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": context deadline exceeded" Nov 02 01:30:05.444318 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:05.424619 4427 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 01:30:05.444318 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:05.424703 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v" podUID="fc786b0e-e9d2-4e73-a48b-42b04f8b3eec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 01:30:05.444318 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:05.427623 4427 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" podUID="366c78ee-4aaf-4b7c-84b4-17028538cb1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:05.768463 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:05.730069 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" podUID="d73ae484-a9b8-4565-a346-0d803a1981ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:05.823483 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:05.789023 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" podUID="47a9069f-1ffa-4c83-be8b-5e62f1d3275a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 02 01:30:05.923234 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:05.896647 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" podUID="cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 02 01:30:05.923234 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:05.914880 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" podUID="eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:05.962287 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:05.935147 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" podUID="f67c954a-ce5b-41d9-8d99-7d126bd89d59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 02 01:30:05.962287 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:05.935676 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" podUID="9c056798-2cf8-47b1-9829-67cebda7a248" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:05.962287 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:05.936010 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" podUID="366c78ee-4aaf-4b7c-84b4-17028538cb1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:05.962287 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:05.936238 4427 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 01:30:05.962287 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:05.936272 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb" podUID="917e0c7d-4eae-4d2d-bb59-f2de47fbfadb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:06.086153 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:06.086108 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" podUID="706df1bd-19cf-4ab9-9aaa-3c12c97ccf45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 02 01:30:06.090139 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:06.090102 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" podUID="fb533b5f-3525-454a-8ce2-ea9246194f28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Nov 02 01:30:06.144922 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:06.142514 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" podUID="ea61971c-5a72-4609-b5a0-58a74b324bb7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:06.164978 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:06.164933 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" podUID="d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Nov 02 01:30:06.191580 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:06.189019 4427 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 02 01:30:06.191580 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:06.189079 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj" podUID="eec37220-0f11-4a42-847f-363344172167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 02 01:30:06.349640 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:06.349534 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" podUID="c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Nov 02 01:30:06.424364 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:06.379533 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" podUID="353a8a41-21ee-4470-861c-b53833beb106" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:06.424840 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:06.424812 4427 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 01:30:06.425004 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:06.424962 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x" podUID="e1412019-62bb-4fc3-a7ad-8e123fef90eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 01:30:06.478407 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:06.425697 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" podUID="7085c0ab-7ee3-4665-abad-2977c25f68ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:06.521437 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:06.426792 4427 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" podUID="353a8a41-21ee-4470-861c-b53833beb106" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:06.521710 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:06.448929 4427 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 01:30:06.521861 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:06.521833 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v" podUID="fc786b0e-e9d2-4e73-a48b-42b04f8b3eec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 01:30:06.733544 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:06.733486 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" podUID="d73ae484-a9b8-4565-a346-0d803a1981ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:06.737735 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:06.737686 4427 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 02 01:30:06.763738 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:06.737932 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj" podUID="eec37220-0f11-4a42-847f-363344172167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 02 01:30:06.921482 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:06.902705 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" podUID="cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:06.921482 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:06.917759 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" podUID="eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:06.940103 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:06.939893 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" podUID="9c056798-2cf8-47b1-9829-67cebda7a248" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 01:30:06.940103 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:06.939999 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" podUID="366c78ee-4aaf-4b7c-84b4-17028538cb1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 02 01:30:06.940103 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:06.939937 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" podUID="f67c954a-ce5b-41d9-8d99-7d126bd89d59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:07.106083 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:07.106039 4427 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" podUID="f67c954a-ce5b-41d9-8d99-7d126bd89d59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:07.106959 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:07.106672 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" podUID="706df1bd-19cf-4ab9-9aaa-3c12c97ccf45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:07.141515 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:07.141475 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" podUID="fb533b5f-3525-454a-8ce2-ea9246194f28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:07.152808 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:07.152709 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" podUID="ea61971c-5a72-4609-b5a0-58a74b324bb7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:07.173609 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:07.172966 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" podUID="d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:07.497330 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:07.469895 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" podUID="c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Nov 02 01:30:07.497330 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:07.472521 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" podUID="353a8a41-21ee-4470-861c-b53833beb106" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:07.622304 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:07.541488 4427 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" podUID="cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:07.622304 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:07.581078 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" podUID="47a9069f-1ffa-4c83-be8b-5e62f1d3275a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:07.622304 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:07.620226 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" podUID="7085c0ab-7ee3-4665-abad-2977c25f68ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:07.942000 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:07.933740 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" podUID="eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:07.942000 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:07.934042 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" podUID="cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:08.055561 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:08.055512 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" podUID="d73ae484-a9b8-4565-a346-0d803a1981ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:08.151484 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:08.128011 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-nddkv" podUID="4379e4b2-e68b-4ddf-a3bf-9ec404e61f56" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.18:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:08.151484 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:30:08.136492 4427 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.197s" Nov 02 01:30:08.151484 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:08.136530 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5687bc-29893050-bvwhp"] Nov 02 01:30:08.151484 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:08.136575 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b6ec07-29893050-x5z5c"] Nov 02 01:30:08.175402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:08.169535 4427 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 01:30:08.175402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:08.169599 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v" podUID="fc786b0e-e9d2-4e73-a48b-42b04f8b3eec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 01:30:08.175402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:08.169716 4427 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 01:30:08.175402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:08.169755 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x" podUID="e1412019-62bb-4fc3-a7ad-8e123fef90eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 01:30:08.175402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:08.169876 4427 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 02 01:30:08.175402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:08.169898 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj" podUID="eec37220-0f11-4a42-847f-363344172167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 02 01:30:08.302399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:08.293614 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-123986-29893050-szk9t"] Nov 02 01:30:08.302399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:08.293817 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" Nov 02 01:30:08.302399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:08.293837 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893050-pbqfk"] Nov 02 01:30:08.403772 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:08.403658 4427 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 01:30:08.404026 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:08.403981 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x" podUID="e1412019-62bb-4fc3-a7ad-8e123fef90eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 01:30:08.460944 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:08.446957 4427 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 01:30:08.460944 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:08.447006 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v" podUID="fc786b0e-e9d2-4e73-a48b-42b04f8b3eec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 01:30:08.687830 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:08.653276 4427 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 02 01:30:08.687830 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:08.653331 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj" podUID="eec37220-0f11-4a42-847f-363344172167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 02 01:30:08.756963 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:08.756934 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" Nov 02 01:30:08.758405 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:08.758361 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" Nov 02 01:30:08.860751 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:08.860415 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq"] Nov 02 01:30:09.052409 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:09.046641 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq"] Nov 02 01:30:09.078402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:09.052773 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq" Nov 02 01:30:09.147689 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:09.147629 4427 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" podUID="fb533b5f-3525-454a-8ce2-ea9246194f28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:09.225029 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:09.224994 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-854ec26e-openshift-default-dockercfg-9x8gl\"" Nov 02 01:30:09.294331 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:09.243459 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79713b28-3c61-4889-a94b-727dc7471195-istio-podinfo\") pod \"e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq\" (UID: \"79713b28-3c61-4889-a94b-727dc7471195\") " pod="openshift-ingress/e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq" Nov 02 01:30:09.294331 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:09.243592 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79713b28-3c61-4889-a94b-727dc7471195-istio-token\") pod \"e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq\" (UID: \"79713b28-3c61-4889-a94b-727dc7471195\") " pod="openshift-ingress/e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq" Nov 02 01:30:09.294331 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:09.243659 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79713b28-3c61-4889-a94b-727dc7471195-workload-socket\") pod \"e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq\" (UID: \"79713b28-3c61-4889-a94b-727dc7471195\") " pod="openshift-ingress/e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq" Nov 02 01:30:09.294331 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:09.243708 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79713b28-3c61-4889-a94b-727dc7471195-workload-certs\") pod \"e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq\" (UID: \"79713b28-3c61-4889-a94b-727dc7471195\") " pod="openshift-ingress/e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq" Nov 02 01:30:09.294331 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:09.243736 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79713b28-3c61-4889-a94b-727dc7471195-istiod-ca-cert\") pod \"e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq\" (UID: \"79713b28-3c61-4889-a94b-727dc7471195\") " pod="openshift-ingress/e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq" Nov 02 01:30:09.294331 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:09.243854 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79713b28-3c61-4889-a94b-727dc7471195-credential-socket\") pod \"e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq\" (UID: \"79713b28-3c61-4889-a94b-727dc7471195\") " pod="openshift-ingress/e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq" Nov 02 01:30:09.294331 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:09.243909 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79713b28-3c61-4889-a94b-727dc7471195-istio-data\") pod \"e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq\" (UID: \"79713b28-3c61-4889-a94b-727dc7471195\") " pod="openshift-ingress/e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq" Nov 02 01:30:09.294331 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:09.244147 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79713b28-3c61-4889-a94b-727dc7471195-istio-envoy\") pod \"e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq\" (UID: \"79713b28-3c61-4889-a94b-727dc7471195\") " pod="openshift-ingress/e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq" Nov 02 01:30:09.294331 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:09.244197 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb792\" (UniqueName: \"kubernetes.io/projected/79713b28-3c61-4889-a94b-727dc7471195-kube-api-access-sb792\") pod \"e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq\" (UID: \"79713b28-3c61-4889-a94b-727dc7471195\") " pod="openshift-ingress/e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq" Nov 02 01:30:09.294331 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:09.245407 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" podUID="9c056798-2cf8-47b1-9829-67cebda7a248" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:09.313020 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:09.295804 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" podUID="d73ae484-a9b8-4565-a346-0d803a1981ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:09.322400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:09.313754 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" podUID="366c78ee-4aaf-4b7c-84b4-17028538cb1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 02 01:30:09.322400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:09.314159 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" podUID="f67c954a-ce5b-41d9-8d99-7d126bd89d59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 02 01:30:09.322400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:09.314474 4427 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" podUID="9c056798-2cf8-47b1-9829-67cebda7a248" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": context deadline exceeded" Nov 02 01:30:09.322400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:09.314531 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" podUID="fb533b5f-3525-454a-8ce2-ea9246194f28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Nov 02 01:30:09.322400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:09.314783 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" podUID="d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Nov 02 01:30:09.322400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:09.315036 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" podUID="c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Nov 02 01:30:09.322400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:09.315288 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" podUID="353a8a41-21ee-4470-861c-b53833beb106" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 02 01:30:09.322400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:09.315561 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" podUID="47a9069f-1ffa-4c83-be8b-5e62f1d3275a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 02 01:30:09.322400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:09.315817 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" podUID="7085c0ab-7ee3-4665-abad-2977c25f68ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Nov 02 01:30:09.432749 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:09.432260 4427 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 01:30:09.432749 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:09.432321 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x" podUID="e1412019-62bb-4fc3-a7ad-8e123fef90eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 01:30:09.483433 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:09.482937 4427 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 01:30:09.526577 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:09.526278 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79713b28-3c61-4889-a94b-727dc7471195-istio-data\") pod \"e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq\" (UID: \"79713b28-3c61-4889-a94b-727dc7471195\") " pod="openshift-ingress/e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq" Nov 02 01:30:09.527661 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:09.527616 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" podUID="cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:09.558700 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:09.483002 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v" podUID="fc786b0e-e9d2-4e73-a48b-42b04f8b3eec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 01:30:09.598871 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:09.594824 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79713b28-3c61-4889-a94b-727dc7471195-istio-data\") pod \"e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq\" (UID: \"79713b28-3c61-4889-a94b-727dc7471195\") " pod="openshift-ingress/e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq" Nov 02 01:30:09.615479 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:09.608985 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79713b28-3c61-4889-a94b-727dc7471195-istio-envoy\") pod \"e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq\" (UID: \"79713b28-3c61-4889-a94b-727dc7471195\") " pod="openshift-ingress/e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq" Nov 02 01:30:09.615479 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:09.609071 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb792\" (UniqueName: \"kubernetes.io/projected/79713b28-3c61-4889-a94b-727dc7471195-kube-api-access-sb792\") pod \"e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq\" (UID: \"79713b28-3c61-4889-a94b-727dc7471195\") " pod="openshift-ingress/e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq" Nov 02 01:30:09.615479 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:09.609147 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79713b28-3c61-4889-a94b-727dc7471195-istio-podinfo\") pod \"e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq\" (UID: \"79713b28-3c61-4889-a94b-727dc7471195\") " pod="openshift-ingress/e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq" Nov 02 01:30:09.615479 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:09.609269 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79713b28-3c61-4889-a94b-727dc7471195-istio-token\") pod \"e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq\" (UID: \"79713b28-3c61-4889-a94b-727dc7471195\") " pod="openshift-ingress/e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq" Nov 02 01:30:09.615479 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:09.609335 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79713b28-3c61-4889-a94b-727dc7471195-workload-socket\") pod \"e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq\" (UID: \"79713b28-3c61-4889-a94b-727dc7471195\") " pod="openshift-ingress/e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq" Nov 02 01:30:09.615479 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:09.609416 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79713b28-3c61-4889-a94b-727dc7471195-workload-certs\") pod \"e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq\" (UID: \"79713b28-3c61-4889-a94b-727dc7471195\") " pod="openshift-ingress/e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq" Nov 02 01:30:09.615479 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:09.609453 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79713b28-3c61-4889-a94b-727dc7471195-istiod-ca-cert\") pod \"e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq\" (UID: \"79713b28-3c61-4889-a94b-727dc7471195\") " pod="openshift-ingress/e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq" Nov 02 01:30:09.615479 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:09.609581 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79713b28-3c61-4889-a94b-727dc7471195-credential-socket\") pod \"e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq\" (UID: \"79713b28-3c61-4889-a94b-727dc7471195\") " pod="openshift-ingress/e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq" Nov 02 01:30:09.615479 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:09.612253 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79713b28-3c61-4889-a94b-727dc7471195-credential-socket\") pod \"e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq\" (UID: \"79713b28-3c61-4889-a94b-727dc7471195\") " pod="openshift-ingress/e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq" Nov 02 01:30:09.635981 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:09.635924 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79713b28-3c61-4889-a94b-727dc7471195-workload-certs\") pod \"e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq\" (UID: \"79713b28-3c61-4889-a94b-727dc7471195\") " pod="openshift-ingress/e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq" Nov 02 01:30:09.636668 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:09.636646 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79713b28-3c61-4889-a94b-727dc7471195-workload-socket\") pod \"e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq\" (UID: \"79713b28-3c61-4889-a94b-727dc7471195\") " pod="openshift-ingress/e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq" Nov 02 01:30:09.664472 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:09.663344 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79713b28-3c61-4889-a94b-727dc7471195-istiod-ca-cert\") pod \"e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq\" (UID: \"79713b28-3c61-4889-a94b-727dc7471195\") " pod="openshift-ingress/e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq" Nov 02 01:30:09.824406 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:09.798509 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" Nov 02 01:30:09.846335 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:09.832966 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79713b28-3c61-4889-a94b-727dc7471195-istio-podinfo\") pod \"e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq\" (UID: \"79713b28-3c61-4889-a94b-727dc7471195\") " pod="openshift-ingress/e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq" Nov 02 01:30:09.918997 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:09.917234 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v" Nov 02 01:30:09.962421 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:09.962246 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb792\" (UniqueName: \"kubernetes.io/projected/79713b28-3c61-4889-a94b-727dc7471195-kube-api-access-sb792\") pod \"e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq\" (UID: \"79713b28-3c61-4889-a94b-727dc7471195\") " pod="openshift-ingress/e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq" Nov 02 01:30:09.978150 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:09.972128 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79713b28-3c61-4889-a94b-727dc7471195-istio-token\") pod \"e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq\" (UID: \"79713b28-3c61-4889-a94b-727dc7471195\") " pod="openshift-ingress/e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq" Nov 02 01:30:09.981414 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:09.981363 4427 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 02 01:30:09.981623 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:09.981603 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj" podUID="eec37220-0f11-4a42-847f-363344172167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 02 01:30:09.982318 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:09.982296 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79713b28-3c61-4889-a94b-727dc7471195-istio-envoy\") pod \"e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq\" (UID: \"79713b28-3c61-4889-a94b-727dc7471195\") " pod="openshift-ingress/e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq" Nov 02 01:30:10.042973 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:10.042944 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x" Nov 02 01:30:10.044043 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:10.044023 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" Nov 02 01:30:10.044204 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:10.044185 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5687bc-29893050-bvwhp" event={"ID":"04f12dc1-5c1e-42ea-a49b-7bc46cf59568","Type":"ContainerStarted","Data":"c81cff199b54d30cebdd1fe6780e47f65e038bdc614b3a5db59bfb81eff94222"} Nov 02 01:30:10.044356 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:10.044344 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" Nov 02 01:30:10.049403 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:10.049359 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893050-pbqfk" event={"ID":"0c82e794-a8fd-4667-8433-c1879a6955b8","Type":"ContainerStarted","Data":"76841d5689e703d6b39fb8601616c3593dbd6e7b39d071f1c81ea9b9f5fb16f1"} Nov 02 01:30:10.049630 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:10.049612 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" Nov 02 01:30:10.049792 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:10.049779 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" Nov 02 01:30:10.049908 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:10.049898 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj" Nov 02 01:30:10.050057 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:10.050046 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" Nov 02 01:30:10.050199 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:10.050189 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" Nov 02 01:30:10.050332 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:10.050322 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" Nov 02 01:30:10.050490 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:10.050477 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" Nov 02 01:30:10.050641 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:10.050631 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" Nov 02 01:30:10.050772 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:10.050762 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" Nov 02 01:30:10.050904 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:10.050894 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" Nov 02 01:30:10.056624 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:10.056589 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq" Nov 02 01:30:10.060131 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:10.060116 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" Nov 02 01:30:10.141976 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:10.141926 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b6ec07-29893050-x5z5c" event={"ID":"76a550cd-124f-45c4-9c6a-77f1eb8f0c97","Type":"ContainerStarted","Data":"e29f8c12ae9e95d76ab47ce024a9dc4c1699039623d4b24448c752a0323235d6"} Nov 02 01:30:10.152696 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:10.152639 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-123986-29893050-szk9t" event={"ID":"9423ce2b-b5fb-4a3b-ae5a-d33802a0a2f8","Type":"ContainerStarted","Data":"4380f53373b48fbd3fb3823780282183ef09b88cc002de81db6f1e056e800752"} Nov 02 01:30:10.399280 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:10.398635 4427 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 01:30:10.401220 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:10.401184 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x" podUID="e1412019-62bb-4fc3-a7ad-8e123fef90eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 01:30:10.403635 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:10.398895 4427 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 01:30:10.403815 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:10.403792 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v" podUID="fc786b0e-e9d2-4e73-a48b-42b04f8b3eec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 01:30:10.817294 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:10.808643 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq"] Nov 02 01:30:10.936477 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:30:10.936433 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79713b28_3c61_4889_a94b_727dc7471195.slice/crio-7b38ae7fb9ce80be3a2a466388d9c73dd98cbb510e6994d44d1696195fd8aebe WatchSource:0}: Error finding container 7b38ae7fb9ce80be3a2a466388d9c73dd98cbb510e6994d44d1696195fd8aebe: Status 404 returned error can't find the container with id 7b38ae7fb9ce80be3a2a466388d9c73dd98cbb510e6994d44d1696195fd8aebe Nov 02 01:30:10.962188 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:10.961658 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:30:10.962188 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:10.961753 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:30:10.962188 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:10.961796 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:30:11.181685 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:11.179461 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5687bc-29893050-bvwhp" event={"ID":"04f12dc1-5c1e-42ea-a49b-7bc46cf59568","Type":"ContainerStarted","Data":"cb081433a1c69dc8c03d8ac2a4f92c5415aee89ff9587148e7125399ddfe09d2"} Nov 02 01:30:11.248401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:11.225525 4427 generic.go:356] "Generic (PLEG): container finished" podID="0c82e794-a8fd-4667-8433-c1879a6955b8" containerID="062d3274222773d9748936ec76bfdb1f1d9ed600d8c7b678808c50ae33e299a4" exitCode=60 Nov 02 01:30:11.248401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:11.225740 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893050-pbqfk" event={"ID":"0c82e794-a8fd-4667-8433-c1879a6955b8","Type":"ContainerDied","Data":"062d3274222773d9748936ec76bfdb1f1d9ed600d8c7b678808c50ae33e299a4"} Nov 02 01:30:11.248401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:11.226164 4427 scope.go:122] "RemoveContainer" containerID="062d3274222773d9748936ec76bfdb1f1d9ed600d8c7b678808c50ae33e299a4" Nov 02 01:30:11.307465 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:11.298058 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5687bc-29893050-bvwhp" podStartSLOduration=11.298037193 podStartE2EDuration="11.298037193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:30:11.222338752 +0000 UTC m=+915.843343989" watchObservedRunningTime="2026-11-02 01:30:11.298037193 +0000 UTC m=+915.919042404" Nov 02 01:30:11.419877 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:11.419778 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq" event={"ID":"79713b28-3c61-4889-a94b-727dc7471195","Type":"ContainerStarted","Data":"7b38ae7fb9ce80be3a2a466388d9c73dd98cbb510e6994d44d1696195fd8aebe"} Nov 02 01:30:11.444817 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:11.444779 4427 generic.go:356] "Generic (PLEG): container finished" podID="76a550cd-124f-45c4-9c6a-77f1eb8f0c97" containerID="a85abeb78e8b7ee6fd5c1dbdc9bee9c45fc9427fcb4a82ed34e5525118fca54e" exitCode=60 Nov 02 01:30:11.445133 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:11.445109 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b6ec07-29893050-x5z5c" event={"ID":"76a550cd-124f-45c4-9c6a-77f1eb8f0c97","Type":"ContainerDied","Data":"a85abeb78e8b7ee6fd5c1dbdc9bee9c45fc9427fcb4a82ed34e5525118fca54e"} Nov 02 01:30:11.445781 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:11.445757 4427 scope.go:122] "RemoveContainer" containerID="a85abeb78e8b7ee6fd5c1dbdc9bee9c45fc9427fcb4a82ed34e5525118fca54e" Nov 02 01:30:11.475841 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:11.466009 4427 generic.go:356] "Generic (PLEG): container finished" podID="9423ce2b-b5fb-4a3b-ae5a-d33802a0a2f8" containerID="527a3f5937ef0d0e6aa6e1ea5788a99a730b003deed16b08b404669c1fc9033c" exitCode=60 Nov 02 01:30:11.475841 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:11.466669 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-123986-29893050-szk9t" event={"ID":"9423ce2b-b5fb-4a3b-ae5a-d33802a0a2f8","Type":"ContainerDied","Data":"527a3f5937ef0d0e6aa6e1ea5788a99a730b003deed16b08b404669c1fc9033c"} Nov 02 01:30:11.475841 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:11.468707 4427 scope.go:122] "RemoveContainer" containerID="527a3f5937ef0d0e6aa6e1ea5788a99a730b003deed16b08b404669c1fc9033c" Nov 02 01:30:11.642081 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:11.641793 4427 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 01:30:11.642081 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:11.641857 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj" podUID="eec37220-0f11-4a42-847f-363344172167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:12.046875 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:12.044331 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v" Nov 02 01:30:12.052066 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:12.051978 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v" Nov 02 01:30:12.375558 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:12.375519 4427 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 01:30:12.379939 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:12.379906 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x" podUID="e1412019-62bb-4fc3-a7ad-8e123fef90eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:12.522422 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:12.521590 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq" event={"ID":"79713b28-3c61-4889-a94b-727dc7471195","Type":"ContainerStarted","Data":"341b2b7bdb462b386a936e655cdc8ce525dbfd0332173cd2a83a96ed281e215a"} Nov 02 01:30:12.530871 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:12.530835 4427 generic.go:356] "Generic (PLEG): container finished" podID="04f12dc1-5c1e-42ea-a49b-7bc46cf59568" containerID="cb081433a1c69dc8c03d8ac2a4f92c5415aee89ff9587148e7125399ddfe09d2" exitCode=60 Nov 02 01:30:12.531045 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:12.530938 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5687bc-29893050-bvwhp" event={"ID":"04f12dc1-5c1e-42ea-a49b-7bc46cf59568","Type":"ContainerDied","Data":"cb081433a1c69dc8c03d8ac2a4f92c5415aee89ff9587148e7125399ddfe09d2"} Nov 02 01:30:12.531446 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:12.531424 4427 scope.go:122] "RemoveContainer" containerID="cb081433a1c69dc8c03d8ac2a4f92c5415aee89ff9587148e7125399ddfe09d2" Nov 02 01:30:12.655327 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:12.653238 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq" podStartSLOduration=4.653215176 podStartE2EDuration="4.653215176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:30:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:30:12.552045384 +0000 UTC m=+917.173050593" watchObservedRunningTime="2026-11-02 01:30:12.653215176 +0000 UTC m=+917.274220390" Nov 02 01:30:12.784319 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:12.784279 4427 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 01:30:12.784618 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:12.784588 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj" podUID="eec37220-0f11-4a42-847f-363344172167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:13.076397 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:13.074459 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq" Nov 02 01:30:13.088831 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:13.088786 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq" Nov 02 01:30:13.280734 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:13.278612 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj" Nov 02 01:30:13.426401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:13.425974 4427 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 01:30:13.426401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:13.426041 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x" podUID="e1412019-62bb-4fc3-a7ad-8e123fef90eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:13.552758 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:13.552667 4427 generic.go:356] "Generic (PLEG): container finished" podID="76a550cd-124f-45c4-9c6a-77f1eb8f0c97" containerID="f71bd6aece777fe3d1ed35e084f4fce9fa262e53f1e8c81bdc185845f31e88e1" exitCode=60 Nov 02 01:30:13.553212 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:13.553158 4427 scope.go:122] "RemoveContainer" containerID="f71bd6aece777fe3d1ed35e084f4fce9fa262e53f1e8c81bdc185845f31e88e1" Nov 02 01:30:13.553653 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:30:13.553623 4427 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b6ec07-29893050-x5z5c_odh-ai-gateway-infra(76a550cd-124f-45c4-9c6a-77f1eb8f0c97)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b6ec07-29893050-x5z5c" podUID="76a550cd-124f-45c4-9c6a-77f1eb8f0c97" Nov 02 01:30:13.556260 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:13.556234 4427 generic.go:356] "Generic (PLEG): container finished" podID="9423ce2b-b5fb-4a3b-ae5a-d33802a0a2f8" containerID="92b980733be7efb166654964723ea5097bee9f47408c942a4ff6ec7cdcff588b" exitCode=60 Nov 02 01:30:13.556760 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:13.556736 4427 scope.go:122] "RemoveContainer" containerID="92b980733be7efb166654964723ea5097bee9f47408c942a4ff6ec7cdcff588b" Nov 02 01:30:13.557180 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:30:13.557137 4427 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-123986-29893050-szk9t_odh-ai-gateway-infra(9423ce2b-b5fb-4a3b-ae5a-d33802a0a2f8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-123986-29893050-szk9t" podUID="9423ce2b-b5fb-4a3b-ae5a-d33802a0a2f8" Nov 02 01:30:13.560133 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:13.560106 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b6ec07-29893050-x5z5c" event={"ID":"76a550cd-124f-45c4-9c6a-77f1eb8f0c97","Type":"ContainerDied","Data":"f71bd6aece777fe3d1ed35e084f4fce9fa262e53f1e8c81bdc185845f31e88e1"} Nov 02 01:30:13.560296 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:13.560278 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-123986-29893050-szk9t" event={"ID":"9423ce2b-b5fb-4a3b-ae5a-d33802a0a2f8","Type":"ContainerDied","Data":"92b980733be7efb166654964723ea5097bee9f47408c942a4ff6ec7cdcff588b"} Nov 02 01:30:13.560543 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:13.560525 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj" Nov 02 01:30:13.560700 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:13.560680 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5687bc-29893050-bvwhp" event={"ID":"04f12dc1-5c1e-42ea-a49b-7bc46cf59568","Type":"ContainerStarted","Data":"143f8bc9762f86fa4bbfa7b178c3a5b0575e980c8b0edb0a1ca6a20e461ceccd"} Nov 02 01:30:13.560847 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:13.560442 4427 scope.go:122] "RemoveContainer" containerID="a85abeb78e8b7ee6fd5c1dbdc9bee9c45fc9427fcb4a82ed34e5525118fca54e" Nov 02 01:30:13.590397 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:13.584605 4427 generic.go:356] "Generic (PLEG): container finished" podID="0c82e794-a8fd-4667-8433-c1879a6955b8" containerID="680b08f992c6b001d295fefb0dd0b238caa940adedb60fc514ecb59901ae9882" exitCode=60 Nov 02 01:30:13.590397 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:13.589958 4427 scope.go:122] "RemoveContainer" containerID="680b08f992c6b001d295fefb0dd0b238caa940adedb60fc514ecb59901ae9882" Nov 02 01:30:13.590397 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:30:13.590361 4427 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893050-pbqfk_odh-ai-gateway-infra(0c82e794-a8fd-4667-8433-c1879a6955b8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893050-pbqfk" podUID="0c82e794-a8fd-4667-8433-c1879a6955b8" Nov 02 01:30:13.590702 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:13.590438 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893050-pbqfk" event={"ID":"0c82e794-a8fd-4667-8433-c1879a6955b8","Type":"ContainerDied","Data":"680b08f992c6b001d295fefb0dd0b238caa940adedb60fc514ecb59901ae9882"} Nov 02 01:30:13.590702 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:13.590490 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq" Nov 02 01:30:13.635250 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:13.634756 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq" Nov 02 01:30:13.645994 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:13.645969 4427 scope.go:122] "RemoveContainer" containerID="527a3f5937ef0d0e6aa6e1ea5788a99a730b003deed16b08b404669c1fc9033c" Nov 02 01:30:13.716797 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:13.702043 4427 scope.go:122] "RemoveContainer" containerID="062d3274222773d9748936ec76bfdb1f1d9ed600d8c7b678808c50ae33e299a4" Nov 02 01:30:14.022055 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:14.019999 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x" Nov 02 01:30:14.022978 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:14.022926 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x" Nov 02 01:30:14.595420 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:14.595353 4427 generic.go:356] "Generic (PLEG): container finished" podID="04f12dc1-5c1e-42ea-a49b-7bc46cf59568" containerID="143f8bc9762f86fa4bbfa7b178c3a5b0575e980c8b0edb0a1ca6a20e461ceccd" exitCode=60 Nov 02 01:30:14.595420 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:14.595407 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5687bc-29893050-bvwhp" event={"ID":"04f12dc1-5c1e-42ea-a49b-7bc46cf59568","Type":"ContainerDied","Data":"143f8bc9762f86fa4bbfa7b178c3a5b0575e980c8b0edb0a1ca6a20e461ceccd"} Nov 02 01:30:14.595981 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:14.595474 4427 scope.go:122] "RemoveContainer" containerID="cb081433a1c69dc8c03d8ac2a4f92c5415aee89ff9587148e7125399ddfe09d2" Nov 02 01:30:14.595981 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:14.595834 4427 scope.go:122] "RemoveContainer" containerID="143f8bc9762f86fa4bbfa7b178c3a5b0575e980c8b0edb0a1ca6a20e461ceccd" Nov 02 01:30:14.596318 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:30:14.596280 4427 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5687bc-29893050-bvwhp_odh-ai-gateway-infra(04f12dc1-5c1e-42ea-a49b-7bc46cf59568)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5687bc-29893050-bvwhp" podUID="04f12dc1-5c1e-42ea-a49b-7bc46cf59568" Nov 02 01:30:14.598585 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:14.598565 4427 scope.go:122] "RemoveContainer" containerID="680b08f992c6b001d295fefb0dd0b238caa940adedb60fc514ecb59901ae9882" Nov 02 01:30:14.598989 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:30:14.598963 4427 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893050-pbqfk_odh-ai-gateway-infra(0c82e794-a8fd-4667-8433-c1879a6955b8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893050-pbqfk" podUID="0c82e794-a8fd-4667-8433-c1879a6955b8" Nov 02 01:30:14.600702 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:14.600684 4427 scope.go:122] "RemoveContainer" containerID="f71bd6aece777fe3d1ed35e084f4fce9fa262e53f1e8c81bdc185845f31e88e1" Nov 02 01:30:14.600990 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:30:14.600966 4427 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b6ec07-29893050-x5z5c_odh-ai-gateway-infra(76a550cd-124f-45c4-9c6a-77f1eb8f0c97)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b6ec07-29893050-x5z5c" podUID="76a550cd-124f-45c4-9c6a-77f1eb8f0c97" Nov 02 01:30:14.602521 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:14.602499 4427 scope.go:122] "RemoveContainer" containerID="92b980733be7efb166654964723ea5097bee9f47408c942a4ff6ec7cdcff588b" Nov 02 01:30:14.602800 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:30:14.602776 4427 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-123986-29893050-szk9t_odh-ai-gateway-infra(9423ce2b-b5fb-4a3b-ae5a-d33802a0a2f8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-123986-29893050-szk9t" podUID="9423ce2b-b5fb-4a3b-ae5a-d33802a0a2f8" Nov 02 01:30:15.669584 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:15.669553 4427 scope.go:122] "RemoveContainer" containerID="143f8bc9762f86fa4bbfa7b178c3a5b0575e980c8b0edb0a1ca6a20e461ceccd" Nov 02 01:30:15.670787 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:30:15.670756 4427 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5687bc-29893050-bvwhp_odh-ai-gateway-infra(04f12dc1-5c1e-42ea-a49b-7bc46cf59568)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5687bc-29893050-bvwhp" podUID="04f12dc1-5c1e-42ea-a49b-7bc46cf59568" Nov 02 01:30:16.417969 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:16.417929 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2f21c877-857c8fbd99-6g6sf"] Nov 02 01:30:16.424872 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:16.424844 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2f21c877-857c8fbd99-6g6sf" Nov 02 01:30:16.430112 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:16.429580 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-2f21c877-dockercfg-tddxz\"" Nov 02 01:30:16.434440 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:16.434406 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2f21c877-857c8fbd99-6g6sf"] Nov 02 01:30:16.441322 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:16.441298 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-2f21c877\"" Nov 02 01:30:16.597940 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:16.597898 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5710831-2b35-4d4a-9ecd-2ea217c8dddd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2f21c877-857c8fbd99-6g6sf\" (UID: \"b5710831-2b35-4d4a-9ecd-2ea217c8dddd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2f21c877-857c8fbd99-6g6sf" Nov 02 01:30:16.598268 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:16.598239 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82pwx\" (UniqueName: \"kubernetes.io/projected/b5710831-2b35-4d4a-9ecd-2ea217c8dddd-kube-api-access-82pwx\") pod \"payload-pre-processing-e2e-ait-2f21c877-857c8fbd99-6g6sf\" (UID: \"b5710831-2b35-4d4a-9ecd-2ea217c8dddd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2f21c877-857c8fbd99-6g6sf" Nov 02 01:30:16.702445 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:16.700560 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5710831-2b35-4d4a-9ecd-2ea217c8dddd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2f21c877-857c8fbd99-6g6sf\" (UID: \"b5710831-2b35-4d4a-9ecd-2ea217c8dddd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2f21c877-857c8fbd99-6g6sf" Nov 02 01:30:16.702445 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:16.700677 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82pwx\" (UniqueName: \"kubernetes.io/projected/b5710831-2b35-4d4a-9ecd-2ea217c8dddd-kube-api-access-82pwx\") pod \"payload-pre-processing-e2e-ait-2f21c877-857c8fbd99-6g6sf\" (UID: \"b5710831-2b35-4d4a-9ecd-2ea217c8dddd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2f21c877-857c8fbd99-6g6sf" Nov 02 01:30:16.702968 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:16.702810 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5710831-2b35-4d4a-9ecd-2ea217c8dddd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2f21c877-857c8fbd99-6g6sf\" (UID: \"b5710831-2b35-4d4a-9ecd-2ea217c8dddd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2f21c877-857c8fbd99-6g6sf" Nov 02 01:30:16.715327 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:16.715295 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82pwx\" (UniqueName: \"kubernetes.io/projected/b5710831-2b35-4d4a-9ecd-2ea217c8dddd-kube-api-access-82pwx\") pod \"payload-pre-processing-e2e-ait-2f21c877-857c8fbd99-6g6sf\" (UID: \"b5710831-2b35-4d4a-9ecd-2ea217c8dddd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2f21c877-857c8fbd99-6g6sf" Nov 02 01:30:16.747364 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:16.747319 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2f21c877-857c8fbd99-6g6sf" Nov 02 01:30:17.109227 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:30:17.109190 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5710831_2b35_4d4a_9ecd_2ea217c8dddd.slice/crio-8c198c8d1384f49c5f8ad7380b015afdbb471ace0229bfcf8bc78d7bbb64b983 WatchSource:0}: Error finding container 8c198c8d1384f49c5f8ad7380b015afdbb471ace0229bfcf8bc78d7bbb64b983: Status 404 returned error can't find the container with id 8c198c8d1384f49c5f8ad7380b015afdbb471ace0229bfcf8bc78d7bbb64b983 Nov 02 01:30:17.120576 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:17.120545 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2f21c877-857c8fbd99-6g6sf"] Nov 02 01:30:17.791402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:17.775620 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2f21c877-857c8fbd99-6g6sf" event={"ID":"b5710831-2b35-4d4a-9ecd-2ea217c8dddd","Type":"ContainerStarted","Data":"97e0ce7b75a6240f97b64af2ece794a0b1cab8941c8ff75fc626f23cfd586d15"} Nov 02 01:30:17.791402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:17.775700 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2f21c877-857c8fbd99-6g6sf" event={"ID":"b5710831-2b35-4d4a-9ecd-2ea217c8dddd","Type":"ContainerStarted","Data":"8c198c8d1384f49c5f8ad7380b015afdbb471ace0229bfcf8bc78d7bbb64b983"} Nov 02 01:30:17.791402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:17.777024 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-2f21c877-857c8fbd99-6g6sf" Nov 02 01:30:17.801609 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:17.800341 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-2f21c877-857c8fbd99-6g6sf" podStartSLOduration=1.800323665 podStartE2EDuration="1.800323665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:30:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:30:17.798756388 +0000 UTC m=+922.419761600" watchObservedRunningTime="2026-11-02 01:30:17.800323665 +0000 UTC m=+922.421328877" Nov 02 01:30:24.149668 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.149633 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-6c45ad5e-575b6cb647-6nqrk"] Nov 02 01:30:24.154661 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.154638 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-6c45ad5e-575b6cb647-6nqrk" Nov 02 01:30:24.164818 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.164790 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-6c45ad5e\"" Nov 02 01:30:24.172708 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.172678 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-6c45ad5e-575b6cb647-6nqrk"] Nov 02 01:30:24.234446 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.234409 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-6c45ad5e-5777dc9557-k6qq2"] Nov 02 01:30:24.253729 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.253695 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-6c45ad5e-5777dc9557-k6qq2" Nov 02 01:30:24.254411 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.254386 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97"] Nov 02 01:30:24.261721 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.261694 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97" Nov 02 01:30:24.262121 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.261601 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-6c45ad5e-5777dc9557-k6qq2"] Nov 02 01:30:24.265318 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.265292 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-6c45ad5e-dockercfg-qs8f7\"" Nov 02 01:30:24.330397 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.322738 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97"] Nov 02 01:30:24.384414 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.382625 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vndbw\" (UniqueName: \"kubernetes.io/projected/31acc518-a706-4c97-ac07-1201d7f58d4d-kube-api-access-vndbw\") pod \"payload-pre-processing-e2e-derive-6c45ad5e-575b6cb647-6nqrk\" (UID: \"31acc518-a706-4c97-ac07-1201d7f58d4d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-6c45ad5e-575b6cb647-6nqrk" Nov 02 01:30:24.384414 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.382984 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31acc518-a706-4c97-ac07-1201d7f58d4d-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-6c45ad5e-575b6cb647-6nqrk\" (UID: \"31acc518-a706-4c97-ac07-1201d7f58d4d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-6c45ad5e-575b6cb647-6nqrk" Nov 02 01:30:24.388408 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.386270 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp"] Nov 02 01:30:24.425411 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.424720 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp" Nov 02 01:30:24.490395 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.487879 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsc2b\" (UniqueName: \"kubernetes.io/projected/1699d426-179c-4719-ac8e-1d05d2e199b8-kube-api-access-gsc2b\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp\" (UID: \"1699d426-179c-4719-ac8e-1d05d2e199b8\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp" Nov 02 01:30:24.490395 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.487956 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31acc518-a706-4c97-ac07-1201d7f58d4d-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-6c45ad5e-575b6cb647-6nqrk\" (UID: \"31acc518-a706-4c97-ac07-1201d7f58d4d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-6c45ad5e-575b6cb647-6nqrk" Nov 02 01:30:24.494457 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.494415 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp"] Nov 02 01:30:24.550512 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.549001 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1699d426-179c-4719-ac8e-1d05d2e199b8-workload-certs\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp\" (UID: \"1699d426-179c-4719-ac8e-1d05d2e199b8\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp" Nov 02 01:30:24.550512 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.549108 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1699d426-179c-4719-ac8e-1d05d2e199b8-workload-socket\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp\" (UID: \"1699d426-179c-4719-ac8e-1d05d2e199b8\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp" Nov 02 01:30:24.550512 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.549156 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c3740bf-b371-41f5-9b96-0096705bf9b9-istiod-ca-cert\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97\" (UID: \"0c3740bf-b371-41f5-9b96-0096705bf9b9\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97" Nov 02 01:30:24.550512 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.549190 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c3740bf-b371-41f5-9b96-0096705bf9b9-workload-certs\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97\" (UID: \"0c3740bf-b371-41f5-9b96-0096705bf9b9\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97" Nov 02 01:30:24.550512 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.549223 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c3740bf-b371-41f5-9b96-0096705bf9b9-istio-podinfo\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97\" (UID: \"0c3740bf-b371-41f5-9b96-0096705bf9b9\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97" Nov 02 01:30:24.550512 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.549288 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7nhm\" (UniqueName: \"kubernetes.io/projected/0c3740bf-b371-41f5-9b96-0096705bf9b9-kube-api-access-f7nhm\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97\" (UID: \"0c3740bf-b371-41f5-9b96-0096705bf9b9\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97" Nov 02 01:30:24.550512 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.549329 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c3740bf-b371-41f5-9b96-0096705bf9b9-istio-token\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97\" (UID: \"0c3740bf-b371-41f5-9b96-0096705bf9b9\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97" Nov 02 01:30:24.550512 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.549365 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vndbw\" (UniqueName: \"kubernetes.io/projected/31acc518-a706-4c97-ac07-1201d7f58d4d-kube-api-access-vndbw\") pod \"payload-pre-processing-e2e-derive-6c45ad5e-575b6cb647-6nqrk\" (UID: \"31acc518-a706-4c97-ac07-1201d7f58d4d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-6c45ad5e-575b6cb647-6nqrk" Nov 02 01:30:24.550512 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.549480 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx8jw\" (UniqueName: \"kubernetes.io/projected/bf344860-07d2-4812-b993-d666d5669910-kube-api-access-rx8jw\") pod \"payload-processing-e2e-derive-6c45ad5e-5777dc9557-k6qq2\" (UID: \"bf344860-07d2-4812-b993-d666d5669910\") " pod="openshift-ingress/payload-processing-e2e-derive-6c45ad5e-5777dc9557-k6qq2" Nov 02 01:30:24.550512 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.549564 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c3740bf-b371-41f5-9b96-0096705bf9b9-credential-socket\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97\" (UID: \"0c3740bf-b371-41f5-9b96-0096705bf9b9\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97" Nov 02 01:30:24.550512 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.549600 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c3740bf-b371-41f5-9b96-0096705bf9b9-istio-envoy\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97\" (UID: \"0c3740bf-b371-41f5-9b96-0096705bf9b9\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97" Nov 02 01:30:24.550512 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.549646 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1699d426-179c-4719-ac8e-1d05d2e199b8-istiod-ca-cert\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp\" (UID: \"1699d426-179c-4719-ac8e-1d05d2e199b8\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp" Nov 02 01:30:24.550512 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.549676 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1699d426-179c-4719-ac8e-1d05d2e199b8-istio-data\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp\" (UID: \"1699d426-179c-4719-ac8e-1d05d2e199b8\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp" Nov 02 01:30:24.550512 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.549817 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1699d426-179c-4719-ac8e-1d05d2e199b8-istio-podinfo\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp\" (UID: \"1699d426-179c-4719-ac8e-1d05d2e199b8\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp" Nov 02 01:30:24.550512 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.549859 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1699d426-179c-4719-ac8e-1d05d2e199b8-credential-socket\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp\" (UID: \"1699d426-179c-4719-ac8e-1d05d2e199b8\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp" Nov 02 01:30:24.550512 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.549883 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1699d426-179c-4719-ac8e-1d05d2e199b8-istio-token\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp\" (UID: \"1699d426-179c-4719-ac8e-1d05d2e199b8\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp" Nov 02 01:30:24.550512 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.549919 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1699d426-179c-4719-ac8e-1d05d2e199b8-istio-envoy\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp\" (UID: \"1699d426-179c-4719-ac8e-1d05d2e199b8\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp" Nov 02 01:30:24.550512 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.549964 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c3740bf-b371-41f5-9b96-0096705bf9b9-istio-data\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97\" (UID: \"0c3740bf-b371-41f5-9b96-0096705bf9b9\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97" Nov 02 01:30:24.550512 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.550005 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c3740bf-b371-41f5-9b96-0096705bf9b9-workload-socket\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97\" (UID: \"0c3740bf-b371-41f5-9b96-0096705bf9b9\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97" Nov 02 01:30:24.550512 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.550079 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf344860-07d2-4812-b993-d666d5669910-plugins-config-volume\") pod \"payload-processing-e2e-derive-6c45ad5e-5777dc9557-k6qq2\" (UID: \"bf344860-07d2-4812-b993-d666d5669910\") " pod="openshift-ingress/payload-processing-e2e-derive-6c45ad5e-5777dc9557-k6qq2" Nov 02 01:30:24.555985 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.555960 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31acc518-a706-4c97-ac07-1201d7f58d4d-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-6c45ad5e-575b6cb647-6nqrk\" (UID: \"31acc518-a706-4c97-ac07-1201d7f58d4d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-6c45ad5e-575b6cb647-6nqrk" Nov 02 01:30:24.627124 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.624359 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vndbw\" (UniqueName: \"kubernetes.io/projected/31acc518-a706-4c97-ac07-1201d7f58d4d-kube-api-access-vndbw\") pod \"payload-pre-processing-e2e-derive-6c45ad5e-575b6cb647-6nqrk\" (UID: \"31acc518-a706-4c97-ac07-1201d7f58d4d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-6c45ad5e-575b6cb647-6nqrk" Nov 02 01:30:24.651584 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.651556 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c3740bf-b371-41f5-9b96-0096705bf9b9-credential-socket\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97\" (UID: \"0c3740bf-b371-41f5-9b96-0096705bf9b9\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97" Nov 02 01:30:24.651834 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.651610 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c3740bf-b371-41f5-9b96-0096705bf9b9-istio-envoy\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97\" (UID: \"0c3740bf-b371-41f5-9b96-0096705bf9b9\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97" Nov 02 01:30:24.651834 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.651644 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1699d426-179c-4719-ac8e-1d05d2e199b8-istiod-ca-cert\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp\" (UID: \"1699d426-179c-4719-ac8e-1d05d2e199b8\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp" Nov 02 01:30:24.651834 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.651684 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1699d426-179c-4719-ac8e-1d05d2e199b8-istio-data\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp\" (UID: \"1699d426-179c-4719-ac8e-1d05d2e199b8\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp" Nov 02 01:30:24.651834 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.651760 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1699d426-179c-4719-ac8e-1d05d2e199b8-istio-podinfo\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp\" (UID: \"1699d426-179c-4719-ac8e-1d05d2e199b8\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp" Nov 02 01:30:24.651834 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.651812 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1699d426-179c-4719-ac8e-1d05d2e199b8-credential-socket\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp\" (UID: \"1699d426-179c-4719-ac8e-1d05d2e199b8\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp" Nov 02 01:30:24.651834 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.651834 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1699d426-179c-4719-ac8e-1d05d2e199b8-istio-token\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp\" (UID: \"1699d426-179c-4719-ac8e-1d05d2e199b8\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp" Nov 02 01:30:24.652113 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.651865 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1699d426-179c-4719-ac8e-1d05d2e199b8-istio-envoy\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp\" (UID: \"1699d426-179c-4719-ac8e-1d05d2e199b8\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp" Nov 02 01:30:24.652113 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.651902 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c3740bf-b371-41f5-9b96-0096705bf9b9-istio-data\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97\" (UID: \"0c3740bf-b371-41f5-9b96-0096705bf9b9\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97" Nov 02 01:30:24.652113 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.651938 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c3740bf-b371-41f5-9b96-0096705bf9b9-workload-socket\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97\" (UID: \"0c3740bf-b371-41f5-9b96-0096705bf9b9\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97" Nov 02 01:30:24.652113 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.651994 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf344860-07d2-4812-b993-d666d5669910-plugins-config-volume\") pod \"payload-processing-e2e-derive-6c45ad5e-5777dc9557-k6qq2\" (UID: \"bf344860-07d2-4812-b993-d666d5669910\") " pod="openshift-ingress/payload-processing-e2e-derive-6c45ad5e-5777dc9557-k6qq2" Nov 02 01:30:24.652113 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.652012 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c3740bf-b371-41f5-9b96-0096705bf9b9-credential-socket\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97\" (UID: \"0c3740bf-b371-41f5-9b96-0096705bf9b9\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97" Nov 02 01:30:24.652113 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.652020 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsc2b\" (UniqueName: \"kubernetes.io/projected/1699d426-179c-4719-ac8e-1d05d2e199b8-kube-api-access-gsc2b\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp\" (UID: \"1699d426-179c-4719-ac8e-1d05d2e199b8\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp" Nov 02 01:30:24.652557 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.652176 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1699d426-179c-4719-ac8e-1d05d2e199b8-workload-certs\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp\" (UID: \"1699d426-179c-4719-ac8e-1d05d2e199b8\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp" Nov 02 01:30:24.652557 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.652293 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1699d426-179c-4719-ac8e-1d05d2e199b8-workload-socket\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp\" (UID: \"1699d426-179c-4719-ac8e-1d05d2e199b8\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp" Nov 02 01:30:24.652557 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.652432 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1699d426-179c-4719-ac8e-1d05d2e199b8-istio-data\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp\" (UID: \"1699d426-179c-4719-ac8e-1d05d2e199b8\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp" Nov 02 01:30:24.652759 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.652642 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1699d426-179c-4719-ac8e-1d05d2e199b8-workload-socket\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp\" (UID: \"1699d426-179c-4719-ac8e-1d05d2e199b8\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp" Nov 02 01:30:24.652853 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.652835 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1699d426-179c-4719-ac8e-1d05d2e199b8-workload-certs\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp\" (UID: \"1699d426-179c-4719-ac8e-1d05d2e199b8\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp" Nov 02 01:30:24.652913 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.652872 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1699d426-179c-4719-ac8e-1d05d2e199b8-istiod-ca-cert\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp\" (UID: \"1699d426-179c-4719-ac8e-1d05d2e199b8\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp" Nov 02 01:30:24.653046 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.653009 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c3740bf-b371-41f5-9b96-0096705bf9b9-workload-socket\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97\" (UID: \"0c3740bf-b371-41f5-9b96-0096705bf9b9\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97" Nov 02 01:30:24.653301 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.653060 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c3740bf-b371-41f5-9b96-0096705bf9b9-istio-data\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97\" (UID: \"0c3740bf-b371-41f5-9b96-0096705bf9b9\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97" Nov 02 01:30:24.653301 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.653099 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c3740bf-b371-41f5-9b96-0096705bf9b9-istiod-ca-cert\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97\" (UID: \"0c3740bf-b371-41f5-9b96-0096705bf9b9\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97" Nov 02 01:30:24.653301 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.653134 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c3740bf-b371-41f5-9b96-0096705bf9b9-workload-certs\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97\" (UID: \"0c3740bf-b371-41f5-9b96-0096705bf9b9\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97" Nov 02 01:30:24.653301 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.653218 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c3740bf-b371-41f5-9b96-0096705bf9b9-istio-podinfo\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97\" (UID: \"0c3740bf-b371-41f5-9b96-0096705bf9b9\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97" Nov 02 01:30:24.653623 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.653283 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7nhm\" (UniqueName: \"kubernetes.io/projected/0c3740bf-b371-41f5-9b96-0096705bf9b9-kube-api-access-f7nhm\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97\" (UID: \"0c3740bf-b371-41f5-9b96-0096705bf9b9\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97" Nov 02 01:30:24.653623 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.653412 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c3740bf-b371-41f5-9b96-0096705bf9b9-istio-token\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97\" (UID: \"0c3740bf-b371-41f5-9b96-0096705bf9b9\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97" Nov 02 01:30:24.653728 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.653672 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf344860-07d2-4812-b993-d666d5669910-plugins-config-volume\") pod \"payload-processing-e2e-derive-6c45ad5e-5777dc9557-k6qq2\" (UID: \"bf344860-07d2-4812-b993-d666d5669910\") " pod="openshift-ingress/payload-processing-e2e-derive-6c45ad5e-5777dc9557-k6qq2" Nov 02 01:30:24.654055 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.654032 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1699d426-179c-4719-ac8e-1d05d2e199b8-credential-socket\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp\" (UID: \"1699d426-179c-4719-ac8e-1d05d2e199b8\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp" Nov 02 01:30:24.655476 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.655449 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c3740bf-b371-41f5-9b96-0096705bf9b9-istio-envoy\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97\" (UID: \"0c3740bf-b371-41f5-9b96-0096705bf9b9\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97" Nov 02 01:30:24.655571 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.655481 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1699d426-179c-4719-ac8e-1d05d2e199b8-istio-envoy\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp\" (UID: \"1699d426-179c-4719-ac8e-1d05d2e199b8\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp" Nov 02 01:30:24.656218 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.656194 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx8jw\" (UniqueName: \"kubernetes.io/projected/bf344860-07d2-4812-b993-d666d5669910-kube-api-access-rx8jw\") pod \"payload-processing-e2e-derive-6c45ad5e-5777dc9557-k6qq2\" (UID: \"bf344860-07d2-4812-b993-d666d5669910\") " pod="openshift-ingress/payload-processing-e2e-derive-6c45ad5e-5777dc9557-k6qq2" Nov 02 01:30:24.656316 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.656225 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c3740bf-b371-41f5-9b96-0096705bf9b9-workload-certs\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97\" (UID: \"0c3740bf-b371-41f5-9b96-0096705bf9b9\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97" Nov 02 01:30:24.656758 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.656739 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c3740bf-b371-41f5-9b96-0096705bf9b9-istiod-ca-cert\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97\" (UID: \"0c3740bf-b371-41f5-9b96-0096705bf9b9\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97" Nov 02 01:30:24.658160 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.658134 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c3740bf-b371-41f5-9b96-0096705bf9b9-istio-podinfo\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97\" (UID: \"0c3740bf-b371-41f5-9b96-0096705bf9b9\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97" Nov 02 01:30:24.658653 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.658636 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1699d426-179c-4719-ac8e-1d05d2e199b8-istio-podinfo\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp\" (UID: \"1699d426-179c-4719-ac8e-1d05d2e199b8\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp" Nov 02 01:30:24.664935 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.664878 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1699d426-179c-4719-ac8e-1d05d2e199b8-istio-token\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp\" (UID: \"1699d426-179c-4719-ac8e-1d05d2e199b8\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp" Nov 02 01:30:24.665057 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.665039 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsc2b\" (UniqueName: \"kubernetes.io/projected/1699d426-179c-4719-ac8e-1d05d2e199b8-kube-api-access-gsc2b\") pod \"e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp\" (UID: \"1699d426-179c-4719-ac8e-1d05d2e199b8\") " pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp" Nov 02 01:30:24.669008 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.668987 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c3740bf-b371-41f5-9b96-0096705bf9b9-istio-token\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97\" (UID: \"0c3740bf-b371-41f5-9b96-0096705bf9b9\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97" Nov 02 01:30:24.670671 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.670501 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7nhm\" (UniqueName: \"kubernetes.io/projected/0c3740bf-b371-41f5-9b96-0096705bf9b9-kube-api-access-f7nhm\") pod \"e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97\" (UID: \"0c3740bf-b371-41f5-9b96-0096705bf9b9\") " pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97" Nov 02 01:30:24.680046 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.680016 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx8jw\" (UniqueName: \"kubernetes.io/projected/bf344860-07d2-4812-b993-d666d5669910-kube-api-access-rx8jw\") pod \"payload-processing-e2e-derive-6c45ad5e-5777dc9557-k6qq2\" (UID: \"bf344860-07d2-4812-b993-d666d5669910\") " pod="openshift-ingress/payload-processing-e2e-derive-6c45ad5e-5777dc9557-k6qq2" Nov 02 01:30:24.695885 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.695820 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-6c45ad5e-5777dc9557-k6qq2" Nov 02 01:30:24.732399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.726672 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97" Nov 02 01:30:24.771299 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.771267 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp" Nov 02 01:30:24.772438 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:24.772413 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-6c45ad5e-575b6cb647-6nqrk" Nov 02 01:30:25.058749 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:25.050555 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-6c45ad5e-5777dc9557-k6qq2"] Nov 02 01:30:25.071152 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:30:25.070959 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf344860_07d2_4812_b993_d666d5669910.slice/crio-be5f356852de029724f8186f606e3a9a1112c7a78f129b0f612131f5435514ef WatchSource:0}: Error finding container be5f356852de029724f8186f606e3a9a1112c7a78f129b0f612131f5435514ef: Status 404 returned error can't find the container with id be5f356852de029724f8186f606e3a9a1112c7a78f129b0f612131f5435514ef Nov 02 01:30:25.162426 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:25.161126 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:30:25.162426 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:25.161280 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:30:25.162426 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:25.161428 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:30:25.193037 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:25.193007 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97"] Nov 02 01:30:25.297783 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:25.295564 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-6c45ad5e-575b6cb647-6nqrk"] Nov 02 01:30:25.363720 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:25.350525 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp"] Nov 02 01:30:25.395407 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:30:25.395323 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1699d426_179c_4719_ac8e_1d05d2e199b8.slice/crio-cfb3378c880b4dbf8ac1fd5f328ec079a5ffd5e7d0c53f5abadec622447d79c5 WatchSource:0}: Error finding container cfb3378c880b4dbf8ac1fd5f328ec079a5ffd5e7d0c53f5abadec622447d79c5: Status 404 returned error can't find the container with id cfb3378c880b4dbf8ac1fd5f328ec079a5ffd5e7d0c53f5abadec622447d79c5 Nov 02 01:30:25.428710 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:25.428630 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:30:25.428921 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:25.428903 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:30:25.429004 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:25.428993 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:30:25.554295 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:25.554268 4427 scope.go:122] "RemoveContainer" containerID="680b08f992c6b001d295fefb0dd0b238caa940adedb60fc514ecb59901ae9882" Nov 02 01:30:25.555713 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:25.555169 4427 scope.go:122] "RemoveContainer" containerID="92b980733be7efb166654964723ea5097bee9f47408c942a4ff6ec7cdcff588b" Nov 02 01:30:25.558111 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:25.556172 4427 scope.go:122] "RemoveContainer" containerID="f71bd6aece777fe3d1ed35e084f4fce9fa262e53f1e8c81bdc185845f31e88e1" Nov 02 01:30:25.963817 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:25.963720 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97" event={"ID":"0c3740bf-b371-41f5-9b96-0096705bf9b9","Type":"ContainerStarted","Data":"22cac3508c764cb692391790ee4b28797982ead11e03bf6b43e6f601bc944360"} Nov 02 01:30:25.963817 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:25.963794 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97" event={"ID":"0c3740bf-b371-41f5-9b96-0096705bf9b9","Type":"ContainerStarted","Data":"b183009234955ccb319309de9c95edd3296a08e98357218671f724d38bf60fa3"} Nov 02 01:30:25.967457 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:25.967423 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-6c45ad5e-5777dc9557-k6qq2" event={"ID":"bf344860-07d2-4812-b993-d666d5669910","Type":"ContainerStarted","Data":"2faf5e8f90b381075b6acbc2f910d94a0b864a5c4aa18a5b4cefeb6dab04e7ca"} Nov 02 01:30:25.967622 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:25.967490 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-6c45ad5e-5777dc9557-k6qq2" event={"ID":"bf344860-07d2-4812-b993-d666d5669910","Type":"ContainerStarted","Data":"be5f356852de029724f8186f606e3a9a1112c7a78f129b0f612131f5435514ef"} Nov 02 01:30:25.967622 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:25.967558 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-6c45ad5e-5777dc9557-k6qq2" Nov 02 01:30:25.974083 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:25.973566 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-6c45ad5e-575b6cb647-6nqrk" event={"ID":"31acc518-a706-4c97-ac07-1201d7f58d4d","Type":"ContainerStarted","Data":"41feff1dbbffcdad1cfa0e796e956b1129a9b3f801ffd3f1d5951c7c25ede75f"} Nov 02 01:30:25.974083 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:25.973634 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-6c45ad5e-575b6cb647-6nqrk" event={"ID":"31acc518-a706-4c97-ac07-1201d7f58d4d","Type":"ContainerStarted","Data":"521bf0d549fe3447324c04726b9ffe0d2e0eeb0589da2e52ba8be4edee607c9b"} Nov 02 01:30:25.975562 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:25.974820 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-6c45ad5e-575b6cb647-6nqrk" Nov 02 01:30:25.987397 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:25.982684 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp" event={"ID":"1699d426-179c-4719-ac8e-1d05d2e199b8","Type":"ContainerStarted","Data":"73180bda1b3e71cfdfd749f16192b9a29a193b44ed72b24f1454fa69a757bf6b"} Nov 02 01:30:25.987397 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:25.982742 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp" event={"ID":"1699d426-179c-4719-ac8e-1d05d2e199b8","Type":"ContainerStarted","Data":"cfb3378c880b4dbf8ac1fd5f328ec079a5ffd5e7d0c53f5abadec622447d79c5"} Nov 02 01:30:26.002144 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:26.002081 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97" podStartSLOduration=2.002060981 podStartE2EDuration="2.002060981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:30:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:30:25.987950926 +0000 UTC m=+930.608956137" watchObservedRunningTime="2026-11-02 01:30:26.002060981 +0000 UTC m=+930.623066216" Nov 02 01:30:26.019569 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:26.019519 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-6c45ad5e-575b6cb647-6nqrk" podStartSLOduration=2.01950199 podStartE2EDuration="2.01950199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:30:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:30:26.016749225 +0000 UTC m=+930.637754438" watchObservedRunningTime="2026-11-02 01:30:26.01950199 +0000 UTC m=+930.640507200" Nov 02 01:30:26.057186 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:26.057126 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-6c45ad5e-5777dc9557-k6qq2" podStartSLOduration=2.057104089 podStartE2EDuration="2.057104089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:30:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:30:26.048455882 +0000 UTC m=+930.669461092" watchObservedRunningTime="2026-11-02 01:30:26.057104089 +0000 UTC m=+930.678109303" Nov 02 01:30:26.077051 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:26.076999 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp" podStartSLOduration=2.076981287 podStartE2EDuration="2.076981287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:30:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:30:26.072160591 +0000 UTC m=+930.693165813" watchObservedRunningTime="2026-11-02 01:30:26.076981287 +0000 UTC m=+930.697986577" Nov 02 01:30:26.738902 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:26.738473 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97" Nov 02 01:30:26.777406 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:26.776882 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp" Nov 02 01:30:26.825098 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:26.825063 4427 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 02 01:30:26.827099 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:26.827055 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp" podUID="1699d426-179c-4719-ac8e-1d05d2e199b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 02 01:30:26.827704 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:26.825340 4427 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 02 01:30:26.827860 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:26.827841 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97" podUID="0c3740bf-b371-41f5-9b96-0096705bf9b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 02 01:30:27.014401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:27.009561 4427 generic.go:356] "Generic (PLEG): container finished" podID="0c82e794-a8fd-4667-8433-c1879a6955b8" containerID="394aaf77ab151205e0bc1e2d53d619bb68ab64c8e00214d4eb38b28731a4d234" exitCode=60 Nov 02 01:30:27.014401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:27.009829 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893050-pbqfk" event={"ID":"0c82e794-a8fd-4667-8433-c1879a6955b8","Type":"ContainerDied","Data":"394aaf77ab151205e0bc1e2d53d619bb68ab64c8e00214d4eb38b28731a4d234"} Nov 02 01:30:27.014401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:27.009896 4427 scope.go:122] "RemoveContainer" containerID="680b08f992c6b001d295fefb0dd0b238caa940adedb60fc514ecb59901ae9882" Nov 02 01:30:27.014401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:27.010501 4427 scope.go:122] "RemoveContainer" containerID="394aaf77ab151205e0bc1e2d53d619bb68ab64c8e00214d4eb38b28731a4d234" Nov 02 01:30:27.014401 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:30:27.010981 4427 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29893050-pbqfk_odh-ai-gateway-infra(0c82e794-a8fd-4667-8433-c1879a6955b8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893050-pbqfk" podUID="0c82e794-a8fd-4667-8433-c1879a6955b8" Nov 02 01:30:27.034393 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:27.034227 4427 generic.go:356] "Generic (PLEG): container finished" podID="76a550cd-124f-45c4-9c6a-77f1eb8f0c97" containerID="f6d09d2a3ef0fb505dd5d30829e02f3612c638b72d346450f54be5d961e4adc9" exitCode=60 Nov 02 01:30:27.034690 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:27.034647 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b6ec07-29893050-x5z5c" event={"ID":"76a550cd-124f-45c4-9c6a-77f1eb8f0c97","Type":"ContainerDied","Data":"f6d09d2a3ef0fb505dd5d30829e02f3612c638b72d346450f54be5d961e4adc9"} Nov 02 01:30:27.035508 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:27.035489 4427 scope.go:122] "RemoveContainer" containerID="f6d09d2a3ef0fb505dd5d30829e02f3612c638b72d346450f54be5d961e4adc9" Nov 02 01:30:27.036165 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:30:27.036128 4427 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b6ec07-29893050-x5z5c_odh-ai-gateway-infra(76a550cd-124f-45c4-9c6a-77f1eb8f0c97)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b6ec07-29893050-x5z5c" podUID="76a550cd-124f-45c4-9c6a-77f1eb8f0c97" Nov 02 01:30:27.051132 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:27.047993 4427 generic.go:356] "Generic (PLEG): container finished" podID="9423ce2b-b5fb-4a3b-ae5a-d33802a0a2f8" containerID="7f10fe6e8432340882e5b8017d3d3ab1a43b3472a669e2df6c15611ea2bd7869" exitCode=60 Nov 02 01:30:27.051132 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:27.049090 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-123986-29893050-szk9t" event={"ID":"9423ce2b-b5fb-4a3b-ae5a-d33802a0a2f8","Type":"ContainerDied","Data":"7f10fe6e8432340882e5b8017d3d3ab1a43b3472a669e2df6c15611ea2bd7869"} Nov 02 01:30:27.051132 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:27.049532 4427 scope.go:122] "RemoveContainer" containerID="7f10fe6e8432340882e5b8017d3d3ab1a43b3472a669e2df6c15611ea2bd7869" Nov 02 01:30:27.051132 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:30:27.049900 4427 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-123986-29893050-szk9t_odh-ai-gateway-infra(9423ce2b-b5fb-4a3b-ae5a-d33802a0a2f8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-123986-29893050-szk9t" podUID="9423ce2b-b5fb-4a3b-ae5a-d33802a0a2f8" Nov 02 01:30:27.106399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:27.105002 4427 scope.go:122] "RemoveContainer" containerID="f71bd6aece777fe3d1ed35e084f4fce9fa262e53f1e8c81bdc185845f31e88e1" Nov 02 01:30:27.158865 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:27.158845 4427 scope.go:122] "RemoveContainer" containerID="92b980733be7efb166654964723ea5097bee9f47408c942a4ff6ec7cdcff588b" Nov 02 01:30:27.531876 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:27.531776 4427 scope.go:122] "RemoveContainer" containerID="143f8bc9762f86fa4bbfa7b178c3a5b0575e980c8b0edb0a1ca6a20e461ceccd" Nov 02 01:30:27.728209 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:27.728169 4427 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 02 01:30:27.728442 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:27.728237 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97" podUID="0c3740bf-b371-41f5-9b96-0096705bf9b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 02 01:30:27.773432 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:27.773166 4427 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 02 01:30:27.773432 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:27.773229 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp" podUID="1699d426-179c-4719-ac8e-1d05d2e199b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 02 01:30:28.046749 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:28.046718 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893050-pbqfk"] Nov 02 01:30:28.154703 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:28.154444 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b6ec07-29893050-x5z5c"] Nov 02 01:30:28.257045 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:28.256081 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-123986-29893050-szk9t"] Nov 02 01:30:28.727767 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:28.727707 4427 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 02 01:30:28.727961 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:28.727777 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97" podUID="0c3740bf-b371-41f5-9b96-0096705bf9b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 02 01:30:28.735132 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:28.735029 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893050-pbqfk" Nov 02 01:30:28.789125 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:28.780529 4427 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 02 01:30:28.789125 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:28.780592 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp" podUID="1699d426-179c-4719-ac8e-1d05d2e199b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 02 01:30:28.850050 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:28.850010 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c82e794-a8fd-4667-8433-c1879a6955b8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c82e794-a8fd-4667-8433-c1879a6955b8-maas-api-service-ca\") pod \"0c82e794-a8fd-4667-8433-c1879a6955b8\" (UID: \"0c82e794-a8fd-4667-8433-c1879a6955b8\") " Nov 02 01:30:28.850335 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:28.850316 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c82e794-a8fd-4667-8433-c1879a6955b8-kube-api-access-xtfcc\" (UniqueName: \"kubernetes.io/projected/0c82e794-a8fd-4667-8433-c1879a6955b8-kube-api-access-xtfcc\") pod \"0c82e794-a8fd-4667-8433-c1879a6955b8\" (UID: \"0c82e794-a8fd-4667-8433-c1879a6955b8\") " Nov 02 01:30:28.852415 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:28.851844 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c82e794-a8fd-4667-8433-c1879a6955b8-maas-api-service-ca" pod "0c82e794-a8fd-4667-8433-c1879a6955b8" (UID: "0c82e794-a8fd-4667-8433-c1879a6955b8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:30:28.855698 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:28.855665 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c82e794-a8fd-4667-8433-c1879a6955b8-kube-api-access-xtfcc" pod "0c82e794-a8fd-4667-8433-c1879a6955b8" (UID: "0c82e794-a8fd-4667-8433-c1879a6955b8"). InnerVolumeSpecName "kube-api-access-xtfcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:30:28.954463 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:28.952626 4427 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c82e794-a8fd-4667-8433-c1879a6955b8-maas-api-service-ca\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:30:28.954463 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:28.952659 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtfcc\" (UniqueName: \"kubernetes.io/projected/0c82e794-a8fd-4667-8433-c1879a6955b8-kube-api-access-xtfcc\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:30:28.964947 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:28.964599 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-123986-29893050-szk9t" Nov 02 01:30:29.031444 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:29.031130 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b6ec07-29893050-x5z5c" Nov 02 01:30:29.110487 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:29.108929 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9423ce2b-b5fb-4a3b-ae5a-d33802a0a2f8-kube-api-access-4k6h7\" (UniqueName: \"kubernetes.io/projected/9423ce2b-b5fb-4a3b-ae5a-d33802a0a2f8-kube-api-access-4k6h7\") pod \"9423ce2b-b5fb-4a3b-ae5a-d33802a0a2f8\" (UID: \"9423ce2b-b5fb-4a3b-ae5a-d33802a0a2f8\") " Nov 02 01:30:29.110487 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:29.109241 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76a550cd-124f-45c4-9c6a-77f1eb8f0c97-kube-api-access-5xkhj\" (UniqueName: \"kubernetes.io/projected/76a550cd-124f-45c4-9c6a-77f1eb8f0c97-kube-api-access-5xkhj\") pod \"76a550cd-124f-45c4-9c6a-77f1eb8f0c97\" (UID: \"76a550cd-124f-45c4-9c6a-77f1eb8f0c97\") " Nov 02 01:30:29.110487 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:29.109292 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76a550cd-124f-45c4-9c6a-77f1eb8f0c97-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76a550cd-124f-45c4-9c6a-77f1eb8f0c97-maas-api-service-ca\") pod \"76a550cd-124f-45c4-9c6a-77f1eb8f0c97\" (UID: \"76a550cd-124f-45c4-9c6a-77f1eb8f0c97\") " Nov 02 01:30:29.111938 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:29.110838 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9423ce2b-b5fb-4a3b-ae5a-d33802a0a2f8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9423ce2b-b5fb-4a3b-ae5a-d33802a0a2f8-maas-api-service-ca\") pod \"9423ce2b-b5fb-4a3b-ae5a-d33802a0a2f8\" (UID: \"9423ce2b-b5fb-4a3b-ae5a-d33802a0a2f8\") " Nov 02 01:30:29.112437 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:29.112189 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9423ce2b-b5fb-4a3b-ae5a-d33802a0a2f8-maas-api-service-ca" pod "9423ce2b-b5fb-4a3b-ae5a-d33802a0a2f8" (UID: "9423ce2b-b5fb-4a3b-ae5a-d33802a0a2f8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:30:29.112933 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:29.112665 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76a550cd-124f-45c4-9c6a-77f1eb8f0c97-maas-api-service-ca" pod "76a550cd-124f-45c4-9c6a-77f1eb8f0c97" (UID: "76a550cd-124f-45c4-9c6a-77f1eb8f0c97"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:30:29.137435 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:29.121866 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76a550cd-124f-45c4-9c6a-77f1eb8f0c97-kube-api-access-5xkhj" pod "76a550cd-124f-45c4-9c6a-77f1eb8f0c97" (UID: "76a550cd-124f-45c4-9c6a-77f1eb8f0c97"). InnerVolumeSpecName "kube-api-access-5xkhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:30:29.137435 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:29.129365 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9423ce2b-b5fb-4a3b-ae5a-d33802a0a2f8-kube-api-access-4k6h7" pod "9423ce2b-b5fb-4a3b-ae5a-d33802a0a2f8" (UID: "9423ce2b-b5fb-4a3b-ae5a-d33802a0a2f8"). InnerVolumeSpecName "kube-api-access-4k6h7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:30:29.147998 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:29.147921 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-123986-29893050-szk9t" event={"ID":"9423ce2b-b5fb-4a3b-ae5a-d33802a0a2f8","Type":"ContainerDied","Data":"4380f53373b48fbd3fb3823780282183ef09b88cc002de81db6f1e056e800752"} Nov 02 01:30:29.147998 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:29.147975 4427 scope.go:122] "RemoveContainer" containerID="7f10fe6e8432340882e5b8017d3d3ab1a43b3472a669e2df6c15611ea2bd7869" Nov 02 01:30:29.148220 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:29.148062 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-123986-29893050-szk9t" Nov 02 01:30:29.153745 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:29.153702 4427 generic.go:356] "Generic (PLEG): container finished" podID="04f12dc1-5c1e-42ea-a49b-7bc46cf59568" containerID="38a0829cbae31407014cfc85b4b4db399b4db35c2ed9e75db2381455d4b5900b" exitCode=60 Nov 02 01:30:29.155151 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:29.153885 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5687bc-29893050-bvwhp" event={"ID":"04f12dc1-5c1e-42ea-a49b-7bc46cf59568","Type":"ContainerDied","Data":"38a0829cbae31407014cfc85b4b4db399b4db35c2ed9e75db2381455d4b5900b"} Nov 02 01:30:29.155151 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:29.154367 4427 scope.go:122] "RemoveContainer" containerID="38a0829cbae31407014cfc85b4b4db399b4db35c2ed9e75db2381455d4b5900b" Nov 02 01:30:29.155151 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:30:29.154908 4427 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5687bc-29893050-bvwhp_odh-ai-gateway-infra(04f12dc1-5c1e-42ea-a49b-7bc46cf59568)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5687bc-29893050-bvwhp" podUID="04f12dc1-5c1e-42ea-a49b-7bc46cf59568" Nov 02 01:30:29.156757 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:29.156113 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893050-pbqfk" Nov 02 01:30:29.158784 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:29.158558 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893050-pbqfk" event={"ID":"0c82e794-a8fd-4667-8433-c1879a6955b8","Type":"ContainerDied","Data":"76841d5689e703d6b39fb8601616c3593dbd6e7b39d071f1c81ea9b9f5fb16f1"} Nov 02 01:30:29.214393 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:29.214283 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xkhj\" (UniqueName: \"kubernetes.io/projected/76a550cd-124f-45c4-9c6a-77f1eb8f0c97-kube-api-access-5xkhj\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:30:29.214393 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:29.214320 4427 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76a550cd-124f-45c4-9c6a-77f1eb8f0c97-maas-api-service-ca\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:30:29.214393 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:29.214336 4427 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9423ce2b-b5fb-4a3b-ae5a-d33802a0a2f8-maas-api-service-ca\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:30:29.214393 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:29.214351 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4k6h7\" (UniqueName: \"kubernetes.io/projected/9423ce2b-b5fb-4a3b-ae5a-d33802a0a2f8-kube-api-access-4k6h7\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:30:29.224917 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:29.224049 4427 scope.go:122] "RemoveContainer" containerID="143f8bc9762f86fa4bbfa7b178c3a5b0575e980c8b0edb0a1ca6a20e461ceccd" Nov 02 01:30:29.224917 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:29.224291 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b6ec07-29893050-x5z5c" event={"ID":"76a550cd-124f-45c4-9c6a-77f1eb8f0c97","Type":"ContainerDied","Data":"e29f8c12ae9e95d76ab47ce024a9dc4c1699039623d4b24448c752a0323235d6"} Nov 02 01:30:29.224917 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:29.224450 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b6ec07-29893050-x5z5c" Nov 02 01:30:29.243669 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:29.243645 4427 scope.go:122] "RemoveContainer" containerID="394aaf77ab151205e0bc1e2d53d619bb68ab64c8e00214d4eb38b28731a4d234" Nov 02 01:30:29.247367 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:29.247340 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-123986-29893050-szk9t"] Nov 02 01:30:29.251524 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:29.251479 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-123986-29893050-szk9t"] Nov 02 01:30:29.255574 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:29.255557 4427 scope.go:122] "RemoveContainer" containerID="f6d09d2a3ef0fb505dd5d30829e02f3612c638b72d346450f54be5d961e4adc9" Nov 02 01:30:29.315737 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:29.315667 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893050-pbqfk"] Nov 02 01:30:29.321259 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:29.321234 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893050-pbqfk"] Nov 02 01:30:29.332130 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:29.332098 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b6ec07-29893050-x5z5c"] Nov 02 01:30:29.336062 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:29.336022 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b6ec07-29893050-x5z5c"] Nov 02 01:30:29.542397 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:29.542302 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c82e794-a8fd-4667-8433-c1879a6955b8" path="/var/lib/kubelet/pods/0c82e794-a8fd-4667-8433-c1879a6955b8/volumes" Nov 02 01:30:29.543136 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:29.543111 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76a550cd-124f-45c4-9c6a-77f1eb8f0c97" path="/var/lib/kubelet/pods/76a550cd-124f-45c4-9c6a-77f1eb8f0c97/volumes" Nov 02 01:30:29.543682 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:29.543665 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9423ce2b-b5fb-4a3b-ae5a-d33802a0a2f8" path="/var/lib/kubelet/pods/9423ce2b-b5fb-4a3b-ae5a-d33802a0a2f8/volumes" Nov 02 01:30:29.728807 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:29.728777 4427 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 02 01:30:29.728955 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:29.728829 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97" podUID="0c3740bf-b371-41f5-9b96-0096705bf9b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 02 01:30:29.779401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:29.778775 4427 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 02 01:30:29.779401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:29.778837 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp" podUID="1699d426-179c-4719-ac8e-1d05d2e199b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 02 01:30:29.848953 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:29.848866 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-2f21c877-857c8fbd99-6g6sf" Nov 02 01:30:30.228244 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:30.228206 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5687bc-29893050-bvwhp"] Nov 02 01:30:30.639756 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:30.639719 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5687bc-29893050-bvwhp" Nov 02 01:30:30.728237 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:30.728198 4427 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 02 01:30:30.728453 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:30.728254 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97" podUID="0c3740bf-b371-41f5-9b96-0096705bf9b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 02 01:30:30.749823 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:30.749795 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04f12dc1-5c1e-42ea-a49b-7bc46cf59568-kube-api-access-sgbmr\" (UniqueName: \"kubernetes.io/projected/04f12dc1-5c1e-42ea-a49b-7bc46cf59568-kube-api-access-sgbmr\") pod \"04f12dc1-5c1e-42ea-a49b-7bc46cf59568\" (UID: \"04f12dc1-5c1e-42ea-a49b-7bc46cf59568\") " Nov 02 01:30:30.750770 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:30.750748 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04f12dc1-5c1e-42ea-a49b-7bc46cf59568-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04f12dc1-5c1e-42ea-a49b-7bc46cf59568-maas-api-service-ca\") pod \"04f12dc1-5c1e-42ea-a49b-7bc46cf59568\" (UID: \"04f12dc1-5c1e-42ea-a49b-7bc46cf59568\") " Nov 02 01:30:30.751423 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:30.751122 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04f12dc1-5c1e-42ea-a49b-7bc46cf59568-maas-api-service-ca" pod "04f12dc1-5c1e-42ea-a49b-7bc46cf59568" (UID: "04f12dc1-5c1e-42ea-a49b-7bc46cf59568"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:30:30.752606 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:30.752588 4427 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04f12dc1-5c1e-42ea-a49b-7bc46cf59568-maas-api-service-ca\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:30:30.814068 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:30.814025 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04f12dc1-5c1e-42ea-a49b-7bc46cf59568-kube-api-access-sgbmr" pod "04f12dc1-5c1e-42ea-a49b-7bc46cf59568" (UID: "04f12dc1-5c1e-42ea-a49b-7bc46cf59568"). InnerVolumeSpecName "kube-api-access-sgbmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:30:30.879408 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:30.874514 4427 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 02 01:30:30.879408 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:30.874576 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp" podUID="1699d426-179c-4719-ac8e-1d05d2e199b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 02 01:30:30.879408 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:30.877116 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgbmr\" (UniqueName: \"kubernetes.io/projected/04f12dc1-5c1e-42ea-a49b-7bc46cf59568-kube-api-access-sgbmr\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:30:31.034918 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:31.032171 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-616e3ed7-bc54d9469-hzxbz"] Nov 02 01:30:31.047940 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:31.047910 4427 cpu_manager.go:414] "containerMap: removing container" podUID="9423ce2b-b5fb-4a3b-ae5a-d33802a0a2f8" containerName="cleanup" Nov 02 01:30:31.048146 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:31.048131 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9423ce2b-b5fb-4a3b-ae5a-d33802a0a2f8" containerName="cleanup" Nov 02 01:30:31.048254 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:31.048244 4427 cpu_manager.go:414] "containerMap: removing container" podUID="0c82e794-a8fd-4667-8433-c1879a6955b8" containerName="cleanup" Nov 02 01:30:31.048336 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:31.048324 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c82e794-a8fd-4667-8433-c1879a6955b8" containerName="cleanup" Nov 02 01:30:31.048464 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:31.048448 4427 cpu_manager.go:414] "containerMap: removing container" podUID="76a550cd-124f-45c4-9c6a-77f1eb8f0c97" containerName="cleanup" Nov 02 01:30:31.048526 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:31.048463 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76a550cd-124f-45c4-9c6a-77f1eb8f0c97" containerName="cleanup" Nov 02 01:30:31.048526 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:31.048477 4427 cpu_manager.go:414] "containerMap: removing container" podUID="76a550cd-124f-45c4-9c6a-77f1eb8f0c97" containerName="cleanup" Nov 02 01:30:31.048526 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:31.048483 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76a550cd-124f-45c4-9c6a-77f1eb8f0c97" containerName="cleanup" Nov 02 01:30:31.048686 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:31.048543 4427 cpu_manager.go:414] "containerMap: removing container" podUID="9423ce2b-b5fb-4a3b-ae5a-d33802a0a2f8" containerName="cleanup" Nov 02 01:30:31.048686 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:31.048552 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9423ce2b-b5fb-4a3b-ae5a-d33802a0a2f8" containerName="cleanup" Nov 02 01:30:31.048686 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:31.048566 4427 cpu_manager.go:414] "containerMap: removing container" podUID="9423ce2b-b5fb-4a3b-ae5a-d33802a0a2f8" containerName="cleanup" Nov 02 01:30:31.048686 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:31.048573 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9423ce2b-b5fb-4a3b-ae5a-d33802a0a2f8" containerName="cleanup" Nov 02 01:30:31.048686 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:31.048598 4427 cpu_manager.go:414] "containerMap: removing container" podUID="04f12dc1-5c1e-42ea-a49b-7bc46cf59568" containerName="cleanup" Nov 02 01:30:31.048686 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:31.048605 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04f12dc1-5c1e-42ea-a49b-7bc46cf59568" containerName="cleanup" Nov 02 01:30:31.048686 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:31.048614 4427 cpu_manager.go:414] "containerMap: removing container" podUID="04f12dc1-5c1e-42ea-a49b-7bc46cf59568" containerName="cleanup" Nov 02 01:30:31.048686 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:31.048621 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04f12dc1-5c1e-42ea-a49b-7bc46cf59568" containerName="cleanup" Nov 02 01:30:31.048686 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:31.048647 4427 cpu_manager.go:414] "containerMap: removing container" podUID="04f12dc1-5c1e-42ea-a49b-7bc46cf59568" containerName="cleanup" Nov 02 01:30:31.048686 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:31.048655 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04f12dc1-5c1e-42ea-a49b-7bc46cf59568" containerName="cleanup" Nov 02 01:30:31.048686 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:31.048687 4427 cpu_manager.go:414] "containerMap: removing container" podUID="0c82e794-a8fd-4667-8433-c1879a6955b8" containerName="cleanup" Nov 02 01:30:31.049171 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:31.048695 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c82e794-a8fd-4667-8433-c1879a6955b8" containerName="cleanup" Nov 02 01:30:31.049171 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:31.048907 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c82e794-a8fd-4667-8433-c1879a6955b8" containerName="cleanup" Nov 02 01:30:31.049171 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:31.048923 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="76a550cd-124f-45c4-9c6a-77f1eb8f0c97" containerName="cleanup" Nov 02 01:30:31.049171 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:31.048933 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="76a550cd-124f-45c4-9c6a-77f1eb8f0c97" containerName="cleanup" Nov 02 01:30:31.049171 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:31.048946 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="9423ce2b-b5fb-4a3b-ae5a-d33802a0a2f8" containerName="cleanup" Nov 02 01:30:31.049171 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:31.048957 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="9423ce2b-b5fb-4a3b-ae5a-d33802a0a2f8" containerName="cleanup" Nov 02 01:30:31.049171 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:31.048971 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="04f12dc1-5c1e-42ea-a49b-7bc46cf59568" containerName="cleanup" Nov 02 01:30:31.049171 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:31.048980 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="04f12dc1-5c1e-42ea-a49b-7bc46cf59568" containerName="cleanup" Nov 02 01:30:31.049171 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:31.049013 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c82e794-a8fd-4667-8433-c1879a6955b8" containerName="cleanup" Nov 02 01:30:31.049171 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:31.049023 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="76a550cd-124f-45c4-9c6a-77f1eb8f0c97" containerName="cleanup" Nov 02 01:30:31.053221 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:31.053198 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-616e3ed7-bc54d9469-hzxbz"] Nov 02 01:30:31.053348 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:31.053317 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-616e3ed7-bc54d9469-hzxbz" Nov 02 01:30:31.093400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:31.085552 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-616e3ed7\"" Nov 02 01:30:31.093400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:31.085701 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-616e3ed7-dockercfg-j2bv9\"" Nov 02 01:30:31.284667 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:31.259730 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ff5b5c-cc7a-4a09-81c8-17987708c2ee-plugins-config-volume\") pod \"payload-processing-e2e-mt-616e3ed7-bc54d9469-hzxbz\" (UID: \"64ff5b5c-cc7a-4a09-81c8-17987708c2ee\") " pod="openshift-ingress/payload-processing-e2e-mt-616e3ed7-bc54d9469-hzxbz" Nov 02 01:30:31.372397 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:31.363897 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2sh6\" (UniqueName: \"kubernetes.io/projected/64ff5b5c-cc7a-4a09-81c8-17987708c2ee-kube-api-access-h2sh6\") pod \"payload-processing-e2e-mt-616e3ed7-bc54d9469-hzxbz\" (UID: \"64ff5b5c-cc7a-4a09-81c8-17987708c2ee\") " pod="openshift-ingress/payload-processing-e2e-mt-616e3ed7-bc54d9469-hzxbz" Nov 02 01:30:31.494180 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:31.494138 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ff5b5c-cc7a-4a09-81c8-17987708c2ee-plugins-config-volume\") pod \"payload-processing-e2e-mt-616e3ed7-bc54d9469-hzxbz\" (UID: \"64ff5b5c-cc7a-4a09-81c8-17987708c2ee\") " pod="openshift-ingress/payload-processing-e2e-mt-616e3ed7-bc54d9469-hzxbz" Nov 02 01:30:31.512564 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:31.506715 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ff5b5c-cc7a-4a09-81c8-17987708c2ee-plugins-config-volume\") pod \"payload-processing-e2e-mt-616e3ed7-bc54d9469-hzxbz\" (UID: \"64ff5b5c-cc7a-4a09-81c8-17987708c2ee\") " pod="openshift-ingress/payload-processing-e2e-mt-616e3ed7-bc54d9469-hzxbz" Nov 02 01:30:31.520078 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:31.519545 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2sh6\" (UniqueName: \"kubernetes.io/projected/64ff5b5c-cc7a-4a09-81c8-17987708c2ee-kube-api-access-h2sh6\") pod \"payload-processing-e2e-mt-616e3ed7-bc54d9469-hzxbz\" (UID: \"64ff5b5c-cc7a-4a09-81c8-17987708c2ee\") " pod="openshift-ingress/payload-processing-e2e-mt-616e3ed7-bc54d9469-hzxbz" Nov 02 01:30:31.549929 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:31.549646 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2sh6\" (UniqueName: \"kubernetes.io/projected/64ff5b5c-cc7a-4a09-81c8-17987708c2ee-kube-api-access-h2sh6\") pod \"payload-processing-e2e-mt-616e3ed7-bc54d9469-hzxbz\" (UID: \"64ff5b5c-cc7a-4a09-81c8-17987708c2ee\") " pod="openshift-ingress/payload-processing-e2e-mt-616e3ed7-bc54d9469-hzxbz" Nov 02 01:30:31.552205 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:31.552180 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5687bc-29893050-bvwhp" Nov 02 01:30:31.559820 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:31.559778 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5687bc-29893050-bvwhp" event={"ID":"04f12dc1-5c1e-42ea-a49b-7bc46cf59568","Type":"ContainerDied","Data":"c81cff199b54d30cebdd1fe6780e47f65e038bdc614b3a5db59bfb81eff94222"} Nov 02 01:30:31.559981 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:31.559846 4427 scope.go:122] "RemoveContainer" containerID="38a0829cbae31407014cfc85b4b4db399b4db35c2ed9e75db2381455d4b5900b" Nov 02 01:30:31.621956 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:30:31.620650 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:30:31.647353 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:31.647319 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5687bc-29893050-bvwhp"] Nov 02 01:30:31.651364 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:31.651338 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5687bc-29893050-bvwhp"] Nov 02 01:30:31.728109 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:31.728063 4427 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 02 01:30:31.728281 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:31.728133 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97" podUID="0c3740bf-b371-41f5-9b96-0096705bf9b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 02 01:30:31.777402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:31.772808 4427 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 02 01:30:31.777402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:31.772876 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp" podUID="1699d426-179c-4719-ac8e-1d05d2e199b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 02 01:30:31.783398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:31.782657 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-616e3ed7-bc54d9469-hzxbz" Nov 02 01:30:32.255563 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:32.255367 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-616e3ed7-bc54d9469-hzxbz"] Nov 02 01:30:32.564870 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:32.564713 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-616e3ed7-bc54d9469-hzxbz" event={"ID":"64ff5b5c-cc7a-4a09-81c8-17987708c2ee","Type":"ContainerStarted","Data":"41709546bc5622c3baa7771c8ef6aa12198e264e1b263931c84f96075e58951b"} Nov 02 01:30:32.564870 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:32.564811 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-616e3ed7-bc54d9469-hzxbz" event={"ID":"64ff5b5c-cc7a-4a09-81c8-17987708c2ee","Type":"ContainerStarted","Data":"3a679fbc3a63988c0dd9e07e92ad8272a6892cf914b051e78f9b6fcc50125f0e"} Nov 02 01:30:32.565237 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:32.565072 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-616e3ed7-bc54d9469-hzxbz" Nov 02 01:30:32.589519 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:32.589444 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-616e3ed7-bc54d9469-hzxbz" podStartSLOduration=2.589428427 podStartE2EDuration="2.589428427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:30:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:30:32.585890936 +0000 UTC m=+937.206896150" watchObservedRunningTime="2026-11-02 01:30:32.589428427 +0000 UTC m=+937.210433638" Nov 02 01:30:32.731462 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:32.730632 4427 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 02 01:30:32.731462 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:32.730729 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97" podUID="0c3740bf-b371-41f5-9b96-0096705bf9b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 02 01:30:32.773398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:32.772871 4427 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 02 01:30:32.773398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:32.772935 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp" podUID="1699d426-179c-4719-ac8e-1d05d2e199b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 02 01:30:33.571399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:33.566241 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04f12dc1-5c1e-42ea-a49b-7bc46cf59568" path="/var/lib/kubelet/pods/04f12dc1-5c1e-42ea-a49b-7bc46cf59568/volumes" Nov 02 01:30:34.718401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:34.714453 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" podUID="fb533b5f-3525-454a-8ce2-ea9246194f28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Nov 02 01:30:34.742553 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:34.736471 4427 patch_prober.go:29] interesting pod/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 01:30:34.742553 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:34.736535 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97" podUID="0c3740bf-b371-41f5-9b96-0096705bf9b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:34.759407 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:34.758598 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" podUID="eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Nov 02 01:30:34.815400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:34.814538 4427 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 01:30:34.815400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:34.814606 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp" podUID="1699d426-179c-4719-ac8e-1d05d2e199b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:34.857400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:34.856488 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" podUID="366c78ee-4aaf-4b7c-84b4-17028538cb1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:35.004597 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:35.004488 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" podUID="7085c0ab-7ee3-4665-abad-2977c25f68ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:35.005278 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:35.005248 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" podUID="cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 02 01:30:35.061492 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:35.061449 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" podUID="47a9069f-1ffa-4c83-be8b-5e62f1d3275a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:35.112521 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:35.112467 4427 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" podUID="7085c0ab-7ee3-4665-abad-2977c25f68ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": context deadline exceeded" Nov 02 01:30:35.112925 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:35.112890 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" podUID="d73ae484-a9b8-4565-a346-0d803a1981ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": context deadline exceeded" Nov 02 01:30:35.138793 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:35.124173 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" podUID="d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:35.164403 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:35.124223 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" podUID="ea61971c-5a72-4609-b5a0-58a74b324bb7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:35.225402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:35.203672 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" podUID="9c056798-2cf8-47b1-9829-67cebda7a248" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:35.331109 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:35.329187 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" podUID="353a8a41-21ee-4470-861c-b53833beb106" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:35.403472 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:35.402536 4427 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" podUID="366c78ee-4aaf-4b7c-84b4-17028538cb1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": context deadline exceeded" Nov 02 01:30:35.473403 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:35.465893 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" podUID="c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:35.504839 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:35.504803 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97" Nov 02 01:30:35.507183 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:35.507158 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97" Nov 02 01:30:35.622879 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:35.622828 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97" Nov 02 01:30:35.750402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:35.720711 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" podUID="fb533b5f-3525-454a-8ce2-ea9246194f28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Nov 02 01:30:35.750402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:35.721552 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" Nov 02 01:30:35.772427 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:35.759461 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" podUID="eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Nov 02 01:30:35.772427 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:35.760206 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" Nov 02 01:30:35.814435 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:35.800637 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" podUID="f67c954a-ce5b-41d9-8d99-7d126bd89d59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:35.869442 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:35.865545 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" podUID="366c78ee-4aaf-4b7c-84b4-17028538cb1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:35.869442 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:35.867691 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" Nov 02 01:30:35.921316 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:35.894174 4427 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 01:30:35.921316 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:35.894234 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp" podUID="1699d426-179c-4719-ac8e-1d05d2e199b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:36.061402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:36.036499 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" podUID="7085c0ab-7ee3-4665-abad-2977c25f68ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Nov 02 01:30:36.071446 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:36.061698 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" podUID="cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 02 01:30:36.092428 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:36.070108 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" Nov 02 01:30:36.092428 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:36.089552 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" Nov 02 01:30:36.224730 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:36.070597 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" podUID="47a9069f-1ffa-4c83-be8b-5e62f1d3275a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:36.225089 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:36.159397 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" podUID="d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:36.225408 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:36.159466 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" podUID="d73ae484-a9b8-4565-a346-0d803a1981ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:36.225879 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:36.184074 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" podUID="ea61971c-5a72-4609-b5a0-58a74b324bb7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:36.226349 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:36.205052 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" podUID="9c056798-2cf8-47b1-9829-67cebda7a248" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:36.283736 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:36.283701 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" Nov 02 01:30:36.287528 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:36.287499 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" Nov 02 01:30:36.291550 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:36.291479 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" Nov 02 01:30:36.384219 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:36.363262 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" podUID="353a8a41-21ee-4470-861c-b53833beb106" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 02 01:30:36.386050 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:36.386028 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" Nov 02 01:30:36.435741 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:36.435005 4427 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" podUID="353a8a41-21ee-4470-861c-b53833beb106" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:36.436836 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:36.436806 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" podUID="706df1bd-19cf-4ab9-9aaa-3c12c97ccf45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:36.437162 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:36.437149 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" Nov 02 01:30:36.447020 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:36.446997 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" Nov 02 01:30:36.486818 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:36.486434 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" podUID="c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:36.492549 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:36.487325 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" Nov 02 01:30:36.813607 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:36.787503 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" podUID="eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:36.833840 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:36.814677 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" podUID="f67c954a-ce5b-41d9-8d99-7d126bd89d59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:36.833840 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:36.832575 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" Nov 02 01:30:36.861150 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:36.845692 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" podUID="fb533b5f-3525-454a-8ce2-ea9246194f28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:36.973681 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:36.933906 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" podUID="366c78ee-4aaf-4b7c-84b4-17028538cb1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 02 01:30:36.991486 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:36.991433 4427 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 01:30:36.991814 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:36.991766 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp" podUID="1699d426-179c-4719-ac8e-1d05d2e199b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:37.167433 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:37.158227 4427 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" podUID="f67c954a-ce5b-41d9-8d99-7d126bd89d59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": context deadline exceeded" Nov 02 01:30:37.168246 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:37.168199 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" podUID="cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:37.222557 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:37.182717 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" podUID="7085c0ab-7ee3-4665-abad-2977c25f68ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:37.242494 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:37.242441 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" podUID="ea61971c-5a72-4609-b5a0-58a74b324bb7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 02 01:30:37.387633 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:37.387587 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" podUID="353a8a41-21ee-4470-861c-b53833beb106" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:37.426458 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:37.400913 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" podUID="d73ae484-a9b8-4565-a346-0d803a1981ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": context deadline exceeded" Nov 02 01:30:37.487681 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:37.443715 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" podUID="706df1bd-19cf-4ab9-9aaa-3c12c97ccf45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:37.517404 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:37.516565 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" Nov 02 01:30:37.601399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:37.600700 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" podUID="c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Nov 02 01:30:37.656661 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:37.656613 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" podUID="d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Nov 02 01:30:37.702949 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:37.702818 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" podUID="9c056798-2cf8-47b1-9829-67cebda7a248" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 01:30:37.812660 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:37.812601 4427 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" podUID="9c056798-2cf8-47b1-9829-67cebda7a248" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:37.821518 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:37.821469 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" podUID="f67c954a-ce5b-41d9-8d99-7d126bd89d59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 02 01:30:37.903396 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:37.899945 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" podUID="eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Nov 02 01:30:38.056461 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:37.949788 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" podUID="47a9069f-1ffa-4c83-be8b-5e62f1d3275a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:38.056461 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:37.949816 4427 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" podUID="cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": context deadline exceeded" Nov 02 01:30:38.056461 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:37.849720 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" podUID="fb533b5f-3525-454a-8ce2-ea9246194f28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:38.086398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:38.074920 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" podUID="366c78ee-4aaf-4b7c-84b4-17028538cb1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:38.164403 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:38.102462 4427 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 01:30:38.164403 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:38.102520 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp" podUID="1699d426-179c-4719-ac8e-1d05d2e199b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:38.201403 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:38.197477 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" podUID="cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:38.236166 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:38.223646 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" podUID="7085c0ab-7ee3-4665-abad-2977c25f68ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Nov 02 01:30:38.236166 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:38.248463 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" podUID="ea61971c-5a72-4609-b5a0-58a74b324bb7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 02 01:30:38.357057 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:38.356895 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-6c45ad5e-5777dc9557-k6qq2" Nov 02 01:30:38.360449 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:38.360421 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-6c45ad5e-575b6cb647-6nqrk" Nov 02 01:30:38.527403 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:38.413490 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" podUID="353a8a41-21ee-4470-861c-b53833beb106" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 02 01:30:38.553397 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:38.463549 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" podUID="d73ae484-a9b8-4565-a346-0d803a1981ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": context deadline exceeded" Nov 02 01:30:38.553397 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:38.463589 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" podUID="706df1bd-19cf-4ab9-9aaa-3c12c97ccf45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 02 01:30:38.625538 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:38.625490 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" podUID="c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:38.746402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:38.722913 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" podUID="9c056798-2cf8-47b1-9829-67cebda7a248" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 01:30:38.747197 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:38.747144 4427 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" podUID="fb533b5f-3525-454a-8ce2-ea9246194f28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": context deadline exceeded" Nov 02 01:30:38.752402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:38.748779 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" podUID="d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Nov 02 01:30:38.855881 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:38.855823 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" podUID="f67c954a-ce5b-41d9-8d99-7d126bd89d59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:38.942300 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:38.942201 4427 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" podUID="c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:38.981573 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:38.981527 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" podUID="47a9069f-1ffa-4c83-be8b-5e62f1d3275a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 02 01:30:38.982222 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:38.982192 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" podUID="eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Nov 02 01:30:39.020792 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:39.020744 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" podUID="fb533b5f-3525-454a-8ce2-ea9246194f28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Nov 02 01:30:39.135782 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:39.097869 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" podUID="366c78ee-4aaf-4b7c-84b4-17028538cb1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:39.209958 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:39.207951 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" podUID="cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 02 01:30:39.295401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:39.242281 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" podUID="7085c0ab-7ee3-4665-abad-2977c25f68ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:39.298010 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:39.260435 4427 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 01:30:39.298755 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:39.298731 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp" podUID="1699d426-179c-4719-ac8e-1d05d2e199b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:39.299161 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:39.298664 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" podUID="ea61971c-5a72-4609-b5a0-58a74b324bb7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:39.555463 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:39.549557 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" podUID="353a8a41-21ee-4470-861c-b53833beb106" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 02 01:30:39.555463 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:39.550153 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" podUID="706df1bd-19cf-4ab9-9aaa-3c12c97ccf45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:39.555463 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:39.550477 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" podUID="d73ae484-a9b8-4565-a346-0d803a1981ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": context deadline exceeded" Nov 02 01:30:39.651219 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:39.651179 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" podUID="c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Nov 02 01:30:39.721742 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:39.721247 4427 patch_prober.go:29] interesting pod/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 01:30:39.721742 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:39.721314 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf" podUID="b880c081-4957-4151-9e1f-fb8f397dedb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:39.767426 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:39.748936 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" podUID="9c056798-2cf8-47b1-9829-67cebda7a248" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:39.796116 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:39.768982 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" podUID="d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Nov 02 01:30:39.796116 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:39.787815 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp" Nov 02 01:30:39.796116 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:39.793966 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" Nov 02 01:30:39.881727 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:39.881463 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" podUID="f67c954a-ce5b-41d9-8d99-7d126bd89d59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:39.985635 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:39.982978 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" podUID="eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:39.993422 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:39.983356 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" podUID="47a9069f-1ffa-4c83-be8b-5e62f1d3275a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 02 01:30:40.101565 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:40.101518 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" podUID="366c78ee-4aaf-4b7c-84b4-17028538cb1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:40.217628 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:40.217528 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" podUID="cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:40.240575 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:40.240520 4427 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" podUID="eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:40.299322 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:40.298201 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" podUID="7085c0ab-7ee3-4665-abad-2977c25f68ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:40.300815 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:40.300753 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" podUID="ea61971c-5a72-4609-b5a0-58a74b324bb7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:40.341364 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:40.340609 4427 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 01:30:40.341364 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:40.340674 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp" podUID="1699d426-179c-4719-ac8e-1d05d2e199b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:40.432716 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:40.432679 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" Nov 02 01:30:40.440683 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:40.440653 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" Nov 02 01:30:40.617524 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:40.616826 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" podUID="d73ae484-a9b8-4565-a346-0d803a1981ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:40.617524 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:40.617218 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" podUID="353a8a41-21ee-4470-861c-b53833beb106" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 02 01:30:40.624395 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:40.617954 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" podUID="706df1bd-19cf-4ab9-9aaa-3c12c97ccf45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 02 01:30:40.649406 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:40.648879 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" Nov 02 01:30:40.653318 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:40.652404 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" podUID="c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:40.674639 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:40.673688 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" Nov 02 01:30:40.885460 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:40.883696 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" podUID="f67c954a-ce5b-41d9-8d99-7d126bd89d59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:40.901404 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:40.901309 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" podUID="d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:41.157281 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:41.145166 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" podUID="9c056798-2cf8-47b1-9829-67cebda7a248" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 01:30:41.209409 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:41.201474 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" podUID="366c78ee-4aaf-4b7c-84b4-17028538cb1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:41.250107 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:41.250075 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" Nov 02 01:30:41.307537 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:41.307487 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" podUID="7085c0ab-7ee3-4665-abad-2977c25f68ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Nov 02 01:30:41.317360 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:41.317318 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" podUID="ea61971c-5a72-4609-b5a0-58a74b324bb7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 02 01:30:41.388664 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:41.388626 4427 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 01:30:41.388999 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:41.388943 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp" podUID="1699d426-179c-4719-ac8e-1d05d2e199b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:41.622406 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:41.621026 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" podUID="353a8a41-21ee-4470-861c-b53833beb106" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:41.622406 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:41.621470 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" podUID="d73ae484-a9b8-4565-a346-0d803a1981ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": context deadline exceeded" Nov 02 01:30:41.662403 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:41.657550 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" podUID="c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:41.855472 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:41.855419 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" podUID="eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:41.912460 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:41.912227 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" podUID="d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Nov 02 01:30:42.226680 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:42.226538 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" podUID="366c78ee-4aaf-4b7c-84b4-17028538cb1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:42.228862 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:42.228827 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" podUID="9c056798-2cf8-47b1-9829-67cebda7a248" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 01:30:42.308087 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:42.308043 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" podUID="7085c0ab-7ee3-4665-abad-2977c25f68ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Nov 02 01:30:42.320431 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:42.319593 4427 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" podUID="d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:42.344501 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:42.344453 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" podUID="ea61971c-5a72-4609-b5a0-58a74b324bb7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:42.456403 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:42.448984 4427 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 01:30:42.456403 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:42.449055 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp" podUID="1699d426-179c-4719-ac8e-1d05d2e199b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:42.509505 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:42.478194 4427 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" podUID="d73ae484-a9b8-4565-a346-0d803a1981ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:42.612535 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:42.612491 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" podUID="706df1bd-19cf-4ab9-9aaa-3c12c97ccf45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:42.654818 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:42.627444 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" podUID="d73ae484-a9b8-4565-a346-0d803a1981ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:42.654818 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:42.644589 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" podUID="353a8a41-21ee-4470-861c-b53833beb106" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:42.659186 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:42.657890 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" podUID="c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:42.873405 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:42.870118 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" podUID="eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:42.896574 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:42.895526 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" podUID="fb533b5f-3525-454a-8ce2-ea9246194f28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:42.911403 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:42.907896 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" Nov 02 01:30:42.922588 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:42.922542 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" podUID="d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:42.923860 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:42.895572 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" podUID="f67c954a-ce5b-41d9-8d99-7d126bd89d59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:43.075993 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:43.075950 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" podUID="47a9069f-1ffa-4c83-be8b-5e62f1d3275a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 02 01:30:43.136489 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:43.118540 4427 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 01:30:43.136489 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:43.118604 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl" podUID="282522e6-7d16-4316-bf07-5e2706155435" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:43.200594 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:43.200531 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" podUID="cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:43.228014 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:43.227966 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" podUID="366c78ee-4aaf-4b7c-84b4-17028538cb1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:43.229310 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:43.229274 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" podUID="9c056798-2cf8-47b1-9829-67cebda7a248" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:43.311696 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:43.310508 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" podUID="7085c0ab-7ee3-4665-abad-2977c25f68ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:43.358488 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:43.348747 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" podUID="ea61971c-5a72-4609-b5a0-58a74b324bb7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:43.462456 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:43.461630 4427 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 01:30:43.462456 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:43.461715 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp" podUID="1699d426-179c-4719-ac8e-1d05d2e199b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:43.620397 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:43.620087 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" podUID="706df1bd-19cf-4ab9-9aaa-3c12c97ccf45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:43.904486 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:43.863772 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" podUID="c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Nov 02 01:30:43.935400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:43.934498 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" podUID="eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:43.935400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:43.934745 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" podUID="fb533b5f-3525-454a-8ce2-ea9246194f28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Nov 02 01:30:43.935400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:43.934891 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" podUID="d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Nov 02 01:30:44.086405 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:44.085549 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" podUID="47a9069f-1ffa-4c83-be8b-5e62f1d3275a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 02 01:30:44.086405 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:44.086010 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" podUID="d73ae484-a9b8-4565-a346-0d803a1981ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": context deadline exceeded" Nov 02 01:30:44.086405 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:44.086077 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" podUID="353a8a41-21ee-4470-861c-b53833beb106" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:44.115516 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:44.115459 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" podUID="f67c954a-ce5b-41d9-8d99-7d126bd89d59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:44.201463 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:44.122045 4427 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" podUID="706df1bd-19cf-4ab9-9aaa-3c12c97ccf45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:44.201463 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:44.138481 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" Nov 02 01:30:44.201463 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:44.138821 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" Nov 02 01:30:44.201463 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:44.139099 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp" Nov 02 01:30:44.201463 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:44.139159 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" Nov 02 01:30:44.201463 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:44.139194 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" Nov 02 01:30:44.201463 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:44.140635 4427 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" podUID="ea61971c-5a72-4609-b5a0-58a74b324bb7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": context deadline exceeded" Nov 02 01:30:44.201463 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:44.140734 4427 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" podUID="47a9069f-1ffa-4c83-be8b-5e62f1d3275a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": context deadline exceeded" Nov 02 01:30:44.276688 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:44.276649 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" podUID="9c056798-2cf8-47b1-9829-67cebda7a248" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:44.336423 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:44.291598 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" podUID="366c78ee-4aaf-4b7c-84b4-17028538cb1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:44.336423 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:44.293045 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp" Nov 02 01:30:44.336423 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:44.319929 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-616e3ed7-bc54d9469-hzxbz" Nov 02 01:30:44.336423 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:44.321902 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" Nov 02 01:30:44.336423 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:44.323976 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" podUID="7085c0ab-7ee3-4665-abad-2977c25f68ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Nov 02 01:30:44.355583 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:44.355539 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" podUID="ea61971c-5a72-4609-b5a0-58a74b324bb7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:44.360427 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:44.358227 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" podUID="cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:44.360427 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:44.358989 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" Nov 02 01:30:44.374587 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:44.371185 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" Nov 02 01:30:44.405847 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:44.394330 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" Nov 02 01:30:44.746899 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:44.743722 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" Nov 02 01:30:44.746899 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:44.744158 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" Nov 02 01:30:44.822563 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:44.821406 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" Nov 02 01:30:44.849879 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:44.849850 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" Nov 02 01:30:44.876419 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:44.875698 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" Nov 02 01:30:44.877402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:44.876770 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" Nov 02 01:30:45.000570 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:45.000275 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" podUID="eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:30:45.004452 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:45.004340 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" Nov 02 01:30:45.018265 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:45.018237 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" Nov 02 01:30:45.018590 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:45.018561 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" Nov 02 01:30:45.019950 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:45.019931 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" Nov 02 01:30:45.024581 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:45.024555 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" Nov 02 01:30:47.563497 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:47.563463 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hwzvb"] Nov 02 01:30:47.564417 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:47.564352 4427 cpu_manager.go:414] "containerMap: removing container" podUID="0c82e794-a8fd-4667-8433-c1879a6955b8" containerName="cleanup" Nov 02 01:30:47.564417 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:47.564392 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c82e794-a8fd-4667-8433-c1879a6955b8" containerName="cleanup" Nov 02 01:30:47.564417 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:47.564416 4427 cpu_manager.go:414] "containerMap: removing container" podUID="76a550cd-124f-45c4-9c6a-77f1eb8f0c97" containerName="cleanup" Nov 02 01:30:47.564645 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:47.564423 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76a550cd-124f-45c4-9c6a-77f1eb8f0c97" containerName="cleanup" Nov 02 01:30:47.564645 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:47.564586 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="9423ce2b-b5fb-4a3b-ae5a-d33802a0a2f8" containerName="cleanup" Nov 02 01:30:47.564645 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:47.564606 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="04f12dc1-5c1e-42ea-a49b-7bc46cf59568" containerName="cleanup" Nov 02 01:30:47.564645 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:47.564622 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c82e794-a8fd-4667-8433-c1879a6955b8" containerName="cleanup" Nov 02 01:30:47.568758 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:47.568737 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hwzvb" Nov 02 01:30:47.577312 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:47.576787 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 02 01:30:47.577312 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:47.576834 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 02 01:30:47.584485 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:47.582018 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 02 01:30:47.584485 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:47.582116 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-tdsjr\"" Nov 02 01:30:47.606492 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:47.606458 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hwzvb"] Nov 02 01:30:47.611967 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:47.608162 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/547c9df8-080d-499b-87d1-69d009750d49-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hwzvb\" (UID: \"547c9df8-080d-499b-87d1-69d009750d49\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hwzvb" Nov 02 01:30:47.611967 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:47.608221 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/547c9df8-080d-499b-87d1-69d009750d49-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hwzvb\" (UID: \"547c9df8-080d-499b-87d1-69d009750d49\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hwzvb" Nov 02 01:30:47.611967 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:47.608309 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/547c9df8-080d-499b-87d1-69d009750d49-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hwzvb\" (UID: \"547c9df8-080d-499b-87d1-69d009750d49\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hwzvb" Nov 02 01:30:47.611967 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:47.608852 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/547c9df8-080d-499b-87d1-69d009750d49-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hwzvb\" (UID: \"547c9df8-080d-499b-87d1-69d009750d49\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hwzvb" Nov 02 01:30:47.611967 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:47.608938 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh9qz\" (UniqueName: \"kubernetes.io/projected/547c9df8-080d-499b-87d1-69d009750d49-kube-api-access-lh9qz\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hwzvb\" (UID: \"547c9df8-080d-499b-87d1-69d009750d49\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hwzvb" Nov 02 01:30:47.611967 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:47.608968 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/547c9df8-080d-499b-87d1-69d009750d49-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hwzvb\" (UID: \"547c9df8-080d-499b-87d1-69d009750d49\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hwzvb" Nov 02 01:30:47.611967 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:47.609004 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/547c9df8-080d-499b-87d1-69d009750d49-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hwzvb\" (UID: \"547c9df8-080d-499b-87d1-69d009750d49\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hwzvb" Nov 02 01:30:47.710246 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:47.710205 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh9qz\" (UniqueName: \"kubernetes.io/projected/547c9df8-080d-499b-87d1-69d009750d49-kube-api-access-lh9qz\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hwzvb\" (UID: \"547c9df8-080d-499b-87d1-69d009750d49\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hwzvb" Nov 02 01:30:47.710429 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:47.710271 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/547c9df8-080d-499b-87d1-69d009750d49-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hwzvb\" (UID: \"547c9df8-080d-499b-87d1-69d009750d49\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hwzvb" Nov 02 01:30:47.710429 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:47.710322 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/547c9df8-080d-499b-87d1-69d009750d49-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hwzvb\" (UID: \"547c9df8-080d-499b-87d1-69d009750d49\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hwzvb" Nov 02 01:30:47.710571 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:47.710448 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/547c9df8-080d-499b-87d1-69d009750d49-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hwzvb\" (UID: \"547c9df8-080d-499b-87d1-69d009750d49\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hwzvb" Nov 02 01:30:47.710571 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:47.710483 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/547c9df8-080d-499b-87d1-69d009750d49-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hwzvb\" (UID: \"547c9df8-080d-499b-87d1-69d009750d49\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hwzvb" Nov 02 01:30:47.710684 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:47.710570 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/547c9df8-080d-499b-87d1-69d009750d49-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hwzvb\" (UID: \"547c9df8-080d-499b-87d1-69d009750d49\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hwzvb" Nov 02 01:30:47.710740 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:47.710693 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/547c9df8-080d-499b-87d1-69d009750d49-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hwzvb\" (UID: \"547c9df8-080d-499b-87d1-69d009750d49\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hwzvb" Nov 02 01:30:47.711646 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:47.711614 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/547c9df8-080d-499b-87d1-69d009750d49-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hwzvb\" (UID: \"547c9df8-080d-499b-87d1-69d009750d49\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hwzvb" Nov 02 01:30:47.713188 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:47.713166 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/547c9df8-080d-499b-87d1-69d009750d49-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hwzvb\" (UID: \"547c9df8-080d-499b-87d1-69d009750d49\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hwzvb" Nov 02 01:30:47.713314 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:47.713291 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/547c9df8-080d-499b-87d1-69d009750d49-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hwzvb\" (UID: \"547c9df8-080d-499b-87d1-69d009750d49\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hwzvb" Nov 02 01:30:47.713395 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:47.713317 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/547c9df8-080d-499b-87d1-69d009750d49-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hwzvb\" (UID: \"547c9df8-080d-499b-87d1-69d009750d49\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hwzvb" Nov 02 01:30:47.713481 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:47.713465 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/547c9df8-080d-499b-87d1-69d009750d49-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hwzvb\" (UID: \"547c9df8-080d-499b-87d1-69d009750d49\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hwzvb" Nov 02 01:30:47.723926 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:47.723905 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh9qz\" (UniqueName: \"kubernetes.io/projected/547c9df8-080d-499b-87d1-69d009750d49-kube-api-access-lh9qz\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hwzvb\" (UID: \"547c9df8-080d-499b-87d1-69d009750d49\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hwzvb" Nov 02 01:30:47.724034 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:47.723960 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/547c9df8-080d-499b-87d1-69d009750d49-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hwzvb\" (UID: \"547c9df8-080d-499b-87d1-69d009750d49\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hwzvb" Nov 02 01:30:47.900353 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:47.900314 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hwzvb" Nov 02 01:30:48.211158 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:48.211128 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hwzvb"] Nov 02 01:30:48.211801 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:30:48.211772 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod547c9df8_080d_499b_87d1_69d009750d49.slice/crio-b7c4e0320896ee27c0fcf80f5bf9f9b391ed1de194597252aa0fdd18cb62a5b6 WatchSource:0}: Error finding container b7c4e0320896ee27c0fcf80f5bf9f9b391ed1de194597252aa0fdd18cb62a5b6: Status 404 returned error can't find the container with id b7c4e0320896ee27c0fcf80f5bf9f9b391ed1de194597252aa0fdd18cb62a5b6 Nov 02 01:30:48.214261 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:48.214243 4427 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 01:30:48.454804 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:48.454770 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hwzvb" event={"ID":"547c9df8-080d-499b-87d1-69d009750d49","Type":"ContainerStarted","Data":"b7c4e0320896ee27c0fcf80f5bf9f9b391ed1de194597252aa0fdd18cb62a5b6"} Nov 02 01:30:51.846570 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:51.846535 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:30:51.846896 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:51.846619 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:30:52.540729 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:52.540690 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hwzvb" event={"ID":"547c9df8-080d-499b-87d1-69d009750d49","Type":"ContainerStarted","Data":"241596ab88afd0104e14692beb122a2a882c9e7a2b15c76ebb6cdd8f9f2401e6"} Nov 02 01:30:52.542092 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:52.542067 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hwzvb" Nov 02 01:30:52.553141 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:52.553096 4427 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-hwzvb container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 01:30:52.553307 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:52.553161 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hwzvb" podUID="547c9df8-080d-499b-87d1-69d009750d49" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 01:30:52.574406 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:52.570323 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hwzvb" podStartSLOduration=1.939849462 podStartE2EDuration="5.570301843s" totalImagesPullingTime="3.630452381s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:30:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:30:52.568243806 +0000 UTC m=+957.189249031" watchObservedRunningTime="2026-11-02 01:30:52.570301843 +0000 UTC m=+957.191307061" Nov 02 01:30:53.561799 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:30:53.561765 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hwzvb" Nov 02 01:31:00.892953 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:00.892768 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2f21c877-857c8fbd99-6g6sf"] Nov 02 01:31:01.589636 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:31:01.589591 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:31:04.584480 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:04.583336 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq"] Nov 02 01:31:04.584480 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:04.583624 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq" podUID="79713b28-3c61-4889-a94b-727dc7471195" containerName="istio-proxy" containerID="cri-o://341b2b7bdb462b386a936e655cdc8ce525dbfd0332173cd2a83a96ed281e215a" gracePeriod=30 Nov 02 01:31:05.893502 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:05.893441 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-2f21c877-857c8fbd99-6g6sf" podUID="b5710831-2b35-4d4a-9ecd-2ea217c8dddd" containerName="payload-pre-processing" containerID="cri-o://97e0ce7b75a6240f97b64af2ece794a0b1cab8941c8ff75fc626f23cfd586d15" gracePeriod=25 Nov 02 01:31:05.995805 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:05.995753 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww"] Nov 02 01:31:06.002310 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:06.002274 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww" Nov 02 01:31:06.015461 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:06.015432 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-9a278165-gw-openshift-default-dockercfg-rnr2w\"" Nov 02 01:31:06.018861 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:06.018794 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww"] Nov 02 01:31:06.122085 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:06.121154 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dldlv\" (UniqueName: \"kubernetes.io/projected/a221bfd5-295a-456b-ae11-afe35719f55d-kube-api-access-dldlv\") pod \"e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww\" (UID: \"a221bfd5-295a-456b-ae11-afe35719f55d\") " pod="openshift-ingress/e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww" Nov 02 01:31:06.122085 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:06.121227 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a221bfd5-295a-456b-ae11-afe35719f55d-istio-envoy\") pod \"e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww\" (UID: \"a221bfd5-295a-456b-ae11-afe35719f55d\") " pod="openshift-ingress/e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww" Nov 02 01:31:06.122085 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:06.121294 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a221bfd5-295a-456b-ae11-afe35719f55d-workload-socket\") pod \"e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww\" (UID: \"a221bfd5-295a-456b-ae11-afe35719f55d\") " pod="openshift-ingress/e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww" Nov 02 01:31:06.122085 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:06.121340 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a221bfd5-295a-456b-ae11-afe35719f55d-workload-certs\") pod \"e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww\" (UID: \"a221bfd5-295a-456b-ae11-afe35719f55d\") " pod="openshift-ingress/e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww" Nov 02 01:31:06.122085 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:06.121434 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a221bfd5-295a-456b-ae11-afe35719f55d-istio-data\") pod \"e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww\" (UID: \"a221bfd5-295a-456b-ae11-afe35719f55d\") " pod="openshift-ingress/e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww" Nov 02 01:31:06.122085 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:06.121511 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a221bfd5-295a-456b-ae11-afe35719f55d-istiod-ca-cert\") pod \"e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww\" (UID: \"a221bfd5-295a-456b-ae11-afe35719f55d\") " pod="openshift-ingress/e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww" Nov 02 01:31:06.122085 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:06.121549 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a221bfd5-295a-456b-ae11-afe35719f55d-istio-token\") pod \"e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww\" (UID: \"a221bfd5-295a-456b-ae11-afe35719f55d\") " pod="openshift-ingress/e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww" Nov 02 01:31:06.122085 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:06.121680 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a221bfd5-295a-456b-ae11-afe35719f55d-credential-socket\") pod \"e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww\" (UID: \"a221bfd5-295a-456b-ae11-afe35719f55d\") " pod="openshift-ingress/e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww" Nov 02 01:31:06.122085 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:06.121722 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a221bfd5-295a-456b-ae11-afe35719f55d-istio-podinfo\") pod \"e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww\" (UID: \"a221bfd5-295a-456b-ae11-afe35719f55d\") " pod="openshift-ingress/e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww" Nov 02 01:31:06.240560 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:06.235989 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a221bfd5-295a-456b-ae11-afe35719f55d-workload-certs\") pod \"e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww\" (UID: \"a221bfd5-295a-456b-ae11-afe35719f55d\") " pod="openshift-ingress/e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww" Nov 02 01:31:06.240560 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:06.236071 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a221bfd5-295a-456b-ae11-afe35719f55d-istio-data\") pod \"e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww\" (UID: \"a221bfd5-295a-456b-ae11-afe35719f55d\") " pod="openshift-ingress/e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww" Nov 02 01:31:06.240560 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:06.236130 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a221bfd5-295a-456b-ae11-afe35719f55d-istiod-ca-cert\") pod \"e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww\" (UID: \"a221bfd5-295a-456b-ae11-afe35719f55d\") " pod="openshift-ingress/e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww" Nov 02 01:31:06.240560 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:06.236168 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a221bfd5-295a-456b-ae11-afe35719f55d-istio-token\") pod \"e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww\" (UID: \"a221bfd5-295a-456b-ae11-afe35719f55d\") " pod="openshift-ingress/e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww" Nov 02 01:31:06.240560 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:06.236287 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a221bfd5-295a-456b-ae11-afe35719f55d-credential-socket\") pod \"e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww\" (UID: \"a221bfd5-295a-456b-ae11-afe35719f55d\") " pod="openshift-ingress/e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww" Nov 02 01:31:06.240560 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:06.236322 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a221bfd5-295a-456b-ae11-afe35719f55d-istio-podinfo\") pod \"e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww\" (UID: \"a221bfd5-295a-456b-ae11-afe35719f55d\") " pod="openshift-ingress/e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww" Nov 02 01:31:06.240560 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:06.236549 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dldlv\" (UniqueName: \"kubernetes.io/projected/a221bfd5-295a-456b-ae11-afe35719f55d-kube-api-access-dldlv\") pod \"e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww\" (UID: \"a221bfd5-295a-456b-ae11-afe35719f55d\") " pod="openshift-ingress/e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww" Nov 02 01:31:06.240560 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:06.236599 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a221bfd5-295a-456b-ae11-afe35719f55d-istio-envoy\") pod \"e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww\" (UID: \"a221bfd5-295a-456b-ae11-afe35719f55d\") " pod="openshift-ingress/e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww" Nov 02 01:31:06.240560 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:06.236661 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a221bfd5-295a-456b-ae11-afe35719f55d-workload-socket\") pod \"e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww\" (UID: \"a221bfd5-295a-456b-ae11-afe35719f55d\") " pod="openshift-ingress/e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww" Nov 02 01:31:06.240560 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:06.237152 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a221bfd5-295a-456b-ae11-afe35719f55d-workload-socket\") pod \"e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww\" (UID: \"a221bfd5-295a-456b-ae11-afe35719f55d\") " pod="openshift-ingress/e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww" Nov 02 01:31:06.240560 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:06.237418 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a221bfd5-295a-456b-ae11-afe35719f55d-workload-certs\") pod \"e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww\" (UID: \"a221bfd5-295a-456b-ae11-afe35719f55d\") " pod="openshift-ingress/e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww" Nov 02 01:31:06.240560 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:06.237662 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a221bfd5-295a-456b-ae11-afe35719f55d-istio-data\") pod \"e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww\" (UID: \"a221bfd5-295a-456b-ae11-afe35719f55d\") " pod="openshift-ingress/e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww" Nov 02 01:31:06.240560 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:06.238288 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a221bfd5-295a-456b-ae11-afe35719f55d-istiod-ca-cert\") pod \"e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww\" (UID: \"a221bfd5-295a-456b-ae11-afe35719f55d\") " pod="openshift-ingress/e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww" Nov 02 01:31:06.269401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:06.248434 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a221bfd5-295a-456b-ae11-afe35719f55d-credential-socket\") pod \"e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww\" (UID: \"a221bfd5-295a-456b-ae11-afe35719f55d\") " pod="openshift-ingress/e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww" Nov 02 01:31:06.325885 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:06.311025 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a221bfd5-295a-456b-ae11-afe35719f55d-istio-envoy\") pod \"e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww\" (UID: \"a221bfd5-295a-456b-ae11-afe35719f55d\") " pod="openshift-ingress/e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww" Nov 02 01:31:06.414434 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:06.409200 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a221bfd5-295a-456b-ae11-afe35719f55d-istio-token\") pod \"e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww\" (UID: \"a221bfd5-295a-456b-ae11-afe35719f55d\") " pod="openshift-ingress/e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww" Nov 02 01:31:06.435258 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:06.435184 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a221bfd5-295a-456b-ae11-afe35719f55d-istio-podinfo\") pod \"e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww\" (UID: \"a221bfd5-295a-456b-ae11-afe35719f55d\") " pod="openshift-ingress/e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww" Nov 02 01:31:06.437551 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:06.436665 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dldlv\" (UniqueName: \"kubernetes.io/projected/a221bfd5-295a-456b-ae11-afe35719f55d-kube-api-access-dldlv\") pod \"e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww\" (UID: \"a221bfd5-295a-456b-ae11-afe35719f55d\") " pod="openshift-ingress/e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww" Nov 02 01:31:06.479104 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:06.479030 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2f21c877-857c8fbd99-6g6sf" Nov 02 01:31:06.507342 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:06.507304 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5710831-2b35-4d4a-9ecd-2ea217c8dddd-kube-api-access-82pwx\" (UniqueName: \"kubernetes.io/projected/b5710831-2b35-4d4a-9ecd-2ea217c8dddd-kube-api-access-82pwx\") pod \"b5710831-2b35-4d4a-9ecd-2ea217c8dddd\" (UID: \"b5710831-2b35-4d4a-9ecd-2ea217c8dddd\") " Nov 02 01:31:06.507568 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:06.507422 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5710831-2b35-4d4a-9ecd-2ea217c8dddd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5710831-2b35-4d4a-9ecd-2ea217c8dddd-plugins-config-volume\") pod \"b5710831-2b35-4d4a-9ecd-2ea217c8dddd\" (UID: \"b5710831-2b35-4d4a-9ecd-2ea217c8dddd\") " Nov 02 01:31:06.507833 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:06.507799 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5710831-2b35-4d4a-9ecd-2ea217c8dddd-plugins-config-volume" pod "b5710831-2b35-4d4a-9ecd-2ea217c8dddd" (UID: "b5710831-2b35-4d4a-9ecd-2ea217c8dddd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:31:06.508314 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:06.508288 4427 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5710831-2b35-4d4a-9ecd-2ea217c8dddd-plugins-config-volume\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:31:06.510553 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:06.510524 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5710831-2b35-4d4a-9ecd-2ea217c8dddd-kube-api-access-82pwx" pod "b5710831-2b35-4d4a-9ecd-2ea217c8dddd" (UID: "b5710831-2b35-4d4a-9ecd-2ea217c8dddd"). InnerVolumeSpecName "kube-api-access-82pwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:31:06.610552 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:06.610508 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82pwx\" (UniqueName: \"kubernetes.io/projected/b5710831-2b35-4d4a-9ecd-2ea217c8dddd-kube-api-access-82pwx\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:31:06.644678 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:06.644643 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww" Nov 02 01:31:06.990819 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:06.990787 4427 generic.go:356] "Generic (PLEG): container finished" podID="b5710831-2b35-4d4a-9ecd-2ea217c8dddd" containerID="97e0ce7b75a6240f97b64af2ece794a0b1cab8941c8ff75fc626f23cfd586d15" exitCode=0 Nov 02 01:31:06.991212 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:06.990837 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2f21c877-857c8fbd99-6g6sf" Nov 02 01:31:06.991212 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:06.990872 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2f21c877-857c8fbd99-6g6sf" event={"ID":"b5710831-2b35-4d4a-9ecd-2ea217c8dddd","Type":"ContainerDied","Data":"97e0ce7b75a6240f97b64af2ece794a0b1cab8941c8ff75fc626f23cfd586d15"} Nov 02 01:31:06.991212 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:06.990940 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2f21c877-857c8fbd99-6g6sf" event={"ID":"b5710831-2b35-4d4a-9ecd-2ea217c8dddd","Type":"ContainerDied","Data":"8c198c8d1384f49c5f8ad7380b015afdbb471ace0229bfcf8bc78d7bbb64b983"} Nov 02 01:31:06.991212 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:06.990965 4427 scope.go:122] "RemoveContainer" containerID="97e0ce7b75a6240f97b64af2ece794a0b1cab8941c8ff75fc626f23cfd586d15" Nov 02 01:31:06.994494 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:06.994469 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww"] Nov 02 01:31:06.996139 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:31:06.995506 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda221bfd5_295a_456b_ae11_afe35719f55d.slice/crio-7cbf2069de9ed8a550fe9ffc697559776ee386b8cff4eef8efd1aeaf8ab07bd7 WatchSource:0}: Error finding container 7cbf2069de9ed8a550fe9ffc697559776ee386b8cff4eef8efd1aeaf8ab07bd7: Status 404 returned error can't find the container with id 7cbf2069de9ed8a550fe9ffc697559776ee386b8cff4eef8efd1aeaf8ab07bd7 Nov 02 01:31:06.998171 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:06.998139 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:31:06.998414 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:06.998341 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:31:06.998534 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:06.998421 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:31:07.112398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:07.111508 4427 scope.go:122] "RemoveContainer" containerID="97e0ce7b75a6240f97b64af2ece794a0b1cab8941c8ff75fc626f23cfd586d15" Nov 02 01:31:07.112398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:07.111627 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2f21c877-857c8fbd99-6g6sf"] Nov 02 01:31:07.137401 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:31:07.120353 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97e0ce7b75a6240f97b64af2ece794a0b1cab8941c8ff75fc626f23cfd586d15\": container with ID starting with 97e0ce7b75a6240f97b64af2ece794a0b1cab8941c8ff75fc626f23cfd586d15 not found: ID does not exist" containerID="97e0ce7b75a6240f97b64af2ece794a0b1cab8941c8ff75fc626f23cfd586d15" Nov 02 01:31:07.137401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:07.120416 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97e0ce7b75a6240f97b64af2ece794a0b1cab8941c8ff75fc626f23cfd586d15"} err="failed to get container status \"97e0ce7b75a6240f97b64af2ece794a0b1cab8941c8ff75fc626f23cfd586d15\": rpc error: code = NotFound desc = could not find container \"97e0ce7b75a6240f97b64af2ece794a0b1cab8941c8ff75fc626f23cfd586d15\": container with ID starting with 97e0ce7b75a6240f97b64af2ece794a0b1cab8941c8ff75fc626f23cfd586d15 not found: ID does not exist" Nov 02 01:31:07.162568 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:07.153097 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2f21c877-857c8fbd99-6g6sf"] Nov 02 01:31:07.485869 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:07.485833 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-6c45ad5e-5777dc9557-k6qq2"] Nov 02 01:31:07.500756 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:07.500161 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-6c45ad5e-575b6cb647-6nqrk"] Nov 02 01:31:07.547468 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:07.543916 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5710831-2b35-4d4a-9ecd-2ea217c8dddd" path="/var/lib/kubelet/pods/b5710831-2b35-4d4a-9ecd-2ea217c8dddd/volumes" Nov 02 01:31:08.034402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:08.033541 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww" event={"ID":"a221bfd5-295a-456b-ae11-afe35719f55d","Type":"ContainerStarted","Data":"da570a30e484ce248c307d15721accd19ef19b786d460e609d87d9272878911d"} Nov 02 01:31:08.034402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:08.033602 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww" event={"ID":"a221bfd5-295a-456b-ae11-afe35719f55d","Type":"ContainerStarted","Data":"7cbf2069de9ed8a550fe9ffc697559776ee386b8cff4eef8efd1aeaf8ab07bd7"} Nov 02 01:31:08.106728 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:08.106664 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww" podStartSLOduration=3.106642611 podStartE2EDuration="3.106642611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:31:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:31:08.083000659 +0000 UTC m=+972.704005871" watchObservedRunningTime="2026-11-02 01:31:08.106642611 +0000 UTC m=+972.727647824" Nov 02 01:31:08.648419 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:08.646987 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww" Nov 02 01:31:08.656177 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:08.654568 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww" Nov 02 01:31:09.061058 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:09.060963 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww" Nov 02 01:31:09.063224 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:09.062599 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww" Nov 02 01:31:10.102636 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:10.102486 4427 generic.go:356] "Generic (PLEG): container finished" podID="79713b28-3c61-4889-a94b-727dc7471195" containerID="341b2b7bdb462b386a936e655cdc8ce525dbfd0332173cd2a83a96ed281e215a" exitCode=0 Nov 02 01:31:10.110111 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:10.103906 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq" event={"ID":"79713b28-3c61-4889-a94b-727dc7471195","Type":"ContainerDied","Data":"341b2b7bdb462b386a936e655cdc8ce525dbfd0332173cd2a83a96ed281e215a"} Nov 02 01:31:10.511033 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:10.511004 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq" Nov 02 01:31:10.609314 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:10.609279 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79713b28-3c61-4889-a94b-727dc7471195-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79713b28-3c61-4889-a94b-727dc7471195-istio-envoy\") pod \"79713b28-3c61-4889-a94b-727dc7471195\" (UID: \"79713b28-3c61-4889-a94b-727dc7471195\") " Nov 02 01:31:10.609536 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:10.609414 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79713b28-3c61-4889-a94b-727dc7471195-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79713b28-3c61-4889-a94b-727dc7471195-workload-socket\") pod \"79713b28-3c61-4889-a94b-727dc7471195\" (UID: \"79713b28-3c61-4889-a94b-727dc7471195\") " Nov 02 01:31:10.609536 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:10.609444 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79713b28-3c61-4889-a94b-727dc7471195-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79713b28-3c61-4889-a94b-727dc7471195-istiod-ca-cert\") pod \"79713b28-3c61-4889-a94b-727dc7471195\" (UID: \"79713b28-3c61-4889-a94b-727dc7471195\") " Nov 02 01:31:10.609536 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:10.609487 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79713b28-3c61-4889-a94b-727dc7471195-istio-token\" (UniqueName: \"kubernetes.io/projected/79713b28-3c61-4889-a94b-727dc7471195-istio-token\") pod \"79713b28-3c61-4889-a94b-727dc7471195\" (UID: \"79713b28-3c61-4889-a94b-727dc7471195\") " Nov 02 01:31:10.609536 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:10.609527 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79713b28-3c61-4889-a94b-727dc7471195-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79713b28-3c61-4889-a94b-727dc7471195-workload-certs\") pod \"79713b28-3c61-4889-a94b-727dc7471195\" (UID: \"79713b28-3c61-4889-a94b-727dc7471195\") " Nov 02 01:31:10.609734 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:10.609625 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79713b28-3c61-4889-a94b-727dc7471195-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79713b28-3c61-4889-a94b-727dc7471195-credential-socket\") pod \"79713b28-3c61-4889-a94b-727dc7471195\" (UID: \"79713b28-3c61-4889-a94b-727dc7471195\") " Nov 02 01:31:10.609734 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:10.609667 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79713b28-3c61-4889-a94b-727dc7471195-kube-api-access-sb792\" (UniqueName: \"kubernetes.io/projected/79713b28-3c61-4889-a94b-727dc7471195-kube-api-access-sb792\") pod \"79713b28-3c61-4889-a94b-727dc7471195\" (UID: \"79713b28-3c61-4889-a94b-727dc7471195\") " Nov 02 01:31:10.609734 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:10.609691 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79713b28-3c61-4889-a94b-727dc7471195-workload-socket" pod "79713b28-3c61-4889-a94b-727dc7471195" (UID: "79713b28-3c61-4889-a94b-727dc7471195"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:31:10.609887 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:10.609785 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79713b28-3c61-4889-a94b-727dc7471195-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79713b28-3c61-4889-a94b-727dc7471195-istio-data\") pod \"79713b28-3c61-4889-a94b-727dc7471195\" (UID: \"79713b28-3c61-4889-a94b-727dc7471195\") " Nov 02 01:31:10.609952 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:10.609877 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79713b28-3c61-4889-a94b-727dc7471195-credential-socket" pod "79713b28-3c61-4889-a94b-727dc7471195" (UID: "79713b28-3c61-4889-a94b-727dc7471195"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:31:10.610008 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:10.609955 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79713b28-3c61-4889-a94b-727dc7471195-istiod-ca-cert" pod "79713b28-3c61-4889-a94b-727dc7471195" (UID: "79713b28-3c61-4889-a94b-727dc7471195"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:31:10.610008 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:10.609964 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79713b28-3c61-4889-a94b-727dc7471195-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79713b28-3c61-4889-a94b-727dc7471195-istio-podinfo\") pod \"79713b28-3c61-4889-a94b-727dc7471195\" (UID: \"79713b28-3c61-4889-a94b-727dc7471195\") " Nov 02 01:31:10.610152 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:10.610111 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79713b28-3c61-4889-a94b-727dc7471195-istio-data" pod "79713b28-3c61-4889-a94b-727dc7471195" (UID: "79713b28-3c61-4889-a94b-727dc7471195"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:31:10.610220 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:10.610145 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79713b28-3c61-4889-a94b-727dc7471195-workload-certs" pod "79713b28-3c61-4889-a94b-727dc7471195" (UID: "79713b28-3c61-4889-a94b-727dc7471195"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:31:10.610940 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:10.610875 4427 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79713b28-3c61-4889-a94b-727dc7471195-istio-data\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:31:10.610940 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:10.610905 4427 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79713b28-3c61-4889-a94b-727dc7471195-workload-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:31:10.610940 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:10.610920 4427 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79713b28-3c61-4889-a94b-727dc7471195-istiod-ca-cert\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:31:10.610940 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:10.610933 4427 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79713b28-3c61-4889-a94b-727dc7471195-workload-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:31:10.611248 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:10.610948 4427 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79713b28-3c61-4889-a94b-727dc7471195-credential-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:31:10.612238 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:10.612208 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79713b28-3c61-4889-a94b-727dc7471195-istio-token" pod "79713b28-3c61-4889-a94b-727dc7471195" (UID: "79713b28-3c61-4889-a94b-727dc7471195"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:31:10.612435 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:10.612415 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79713b28-3c61-4889-a94b-727dc7471195-kube-api-access-sb792" pod "79713b28-3c61-4889-a94b-727dc7471195" (UID: "79713b28-3c61-4889-a94b-727dc7471195"). InnerVolumeSpecName "kube-api-access-sb792". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:31:10.612770 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:10.612743 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79713b28-3c61-4889-a94b-727dc7471195-istio-podinfo" pod "79713b28-3c61-4889-a94b-727dc7471195" (UID: "79713b28-3c61-4889-a94b-727dc7471195"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:31:10.612770 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:10.612761 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79713b28-3c61-4889-a94b-727dc7471195-istio-envoy" pod "79713b28-3c61-4889-a94b-727dc7471195" (UID: "79713b28-3c61-4889-a94b-727dc7471195"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:31:10.712473 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:10.712439 4427 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79713b28-3c61-4889-a94b-727dc7471195-istio-podinfo\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:31:10.712473 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:10.712468 4427 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79713b28-3c61-4889-a94b-727dc7471195-istio-envoy\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:31:10.712473 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:10.712478 4427 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79713b28-3c61-4889-a94b-727dc7471195-istio-token\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:31:10.712765 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:10.712489 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb792\" (UniqueName: \"kubernetes.io/projected/79713b28-3c61-4889-a94b-727dc7471195-kube-api-access-sb792\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:31:10.756568 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:10.756533 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9a278165-5848cdd479-vq67l"] Nov 02 01:31:10.757503 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:10.757485 4427 cpu_manager.go:414] "containerMap: removing container" podUID="b5710831-2b35-4d4a-9ecd-2ea217c8dddd" containerName="payload-pre-processing" Nov 02 01:31:10.757572 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:10.757506 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5710831-2b35-4d4a-9ecd-2ea217c8dddd" containerName="payload-pre-processing" Nov 02 01:31:10.757572 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:10.757532 4427 cpu_manager.go:414] "containerMap: removing container" podUID="79713b28-3c61-4889-a94b-727dc7471195" containerName="istio-proxy" Nov 02 01:31:10.757572 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:10.757540 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79713b28-3c61-4889-a94b-727dc7471195" containerName="istio-proxy" Nov 02 01:31:10.757731 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:10.757720 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="79713b28-3c61-4889-a94b-727dc7471195" containerName="istio-proxy" Nov 02 01:31:10.757769 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:10.757749 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5710831-2b35-4d4a-9ecd-2ea217c8dddd" containerName="payload-pre-processing" Nov 02 01:31:10.762295 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:10.762268 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9a278165-5848cdd479-vq67l" Nov 02 01:31:10.774466 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:10.770662 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-9a278165-dockercfg-j8v4g\"" Nov 02 01:31:10.774466 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:10.773728 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9a278165-5848cdd479-vq67l"] Nov 02 01:31:10.845696 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:10.845666 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-9a278165\"" Nov 02 01:31:10.947451 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:10.932273 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmsbv\" (UniqueName: \"kubernetes.io/projected/4e04e863-2c20-4c9e-91e2-2c6ecf4bb627-kube-api-access-hmsbv\") pod \"payload-pre-processing-e2e-migrate-9a278165-5848cdd479-vq67l\" (UID: \"4e04e863-2c20-4c9e-91e2-2c6ecf4bb627\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9a278165-5848cdd479-vq67l" Nov 02 01:31:10.947451 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:10.932523 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e04e863-2c20-4c9e-91e2-2c6ecf4bb627-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-9a278165-5848cdd479-vq67l\" (UID: \"4e04e863-2c20-4c9e-91e2-2c6ecf4bb627\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9a278165-5848cdd479-vq67l" Nov 02 01:31:11.039399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:11.038299 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmsbv\" (UniqueName: \"kubernetes.io/projected/4e04e863-2c20-4c9e-91e2-2c6ecf4bb627-kube-api-access-hmsbv\") pod \"payload-pre-processing-e2e-migrate-9a278165-5848cdd479-vq67l\" (UID: \"4e04e863-2c20-4c9e-91e2-2c6ecf4bb627\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9a278165-5848cdd479-vq67l" Nov 02 01:31:11.061581 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:11.059469 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e04e863-2c20-4c9e-91e2-2c6ecf4bb627-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-9a278165-5848cdd479-vq67l\" (UID: \"4e04e863-2c20-4c9e-91e2-2c6ecf4bb627\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9a278165-5848cdd479-vq67l" Nov 02 01:31:11.062533 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:11.062428 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e04e863-2c20-4c9e-91e2-2c6ecf4bb627-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-9a278165-5848cdd479-vq67l\" (UID: \"4e04e863-2c20-4c9e-91e2-2c6ecf4bb627\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9a278165-5848cdd479-vq67l" Nov 02 01:31:11.148965 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:11.148424 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmsbv\" (UniqueName: \"kubernetes.io/projected/4e04e863-2c20-4c9e-91e2-2c6ecf4bb627-kube-api-access-hmsbv\") pod \"payload-pre-processing-e2e-migrate-9a278165-5848cdd479-vq67l\" (UID: \"4e04e863-2c20-4c9e-91e2-2c6ecf4bb627\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9a278165-5848cdd479-vq67l" Nov 02 01:31:11.156574 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:11.155990 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq" event={"ID":"79713b28-3c61-4889-a94b-727dc7471195","Type":"ContainerDied","Data":"7b38ae7fb9ce80be3a2a466388d9c73dd98cbb510e6994d44d1696195fd8aebe"} Nov 02 01:31:11.156574 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:11.156056 4427 scope.go:122] "RemoveContainer" containerID="341b2b7bdb462b386a936e655cdc8ce525dbfd0332173cd2a83a96ed281e215a" Nov 02 01:31:11.156574 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:11.156200 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq" Nov 02 01:31:11.187315 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:11.187282 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq"] Nov 02 01:31:11.191607 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:11.191581 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-854ec26e-openshift-default-6cdf65c89c-c9nrq"] Nov 02 01:31:11.386973 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:11.386939 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9a278165-5848cdd479-vq67l" Nov 02 01:31:11.615823 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:11.595688 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79713b28-3c61-4889-a94b-727dc7471195" path="/var/lib/kubelet/pods/79713b28-3c61-4889-a94b-727dc7471195/volumes" Nov 02 01:31:11.819296 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:31:11.819263 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e04e863_2c20_4c9e_91e2_2c6ecf4bb627.slice/crio-22fc8766d83d1a1e44f28f5f6dce14207b00179dcf85ea035912489aba994af8 WatchSource:0}: Error finding container 22fc8766d83d1a1e44f28f5f6dce14207b00179dcf85ea035912489aba994af8: Status 404 returned error can't find the container with id 22fc8766d83d1a1e44f28f5f6dce14207b00179dcf85ea035912489aba994af8 Nov 02 01:31:11.819678 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:11.819652 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9a278165-5848cdd479-vq67l"] Nov 02 01:31:12.259786 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:12.258907 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9a278165-5848cdd479-vq67l" event={"ID":"4e04e863-2c20-4c9e-91e2-2c6ecf4bb627","Type":"ContainerStarted","Data":"437bc0cea8cb431dd25836fe9aa303872722bbbd43fd1d2bc7a7ec3e2ae35126"} Nov 02 01:31:12.259786 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:12.258976 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9a278165-5848cdd479-vq67l" event={"ID":"4e04e863-2c20-4c9e-91e2-2c6ecf4bb627","Type":"ContainerStarted","Data":"22fc8766d83d1a1e44f28f5f6dce14207b00179dcf85ea035912489aba994af8"} Nov 02 01:31:12.260404 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:12.260179 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9a278165-5848cdd479-vq67l" Nov 02 01:31:12.486228 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:12.486172 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-6c45ad5e-5777dc9557-k6qq2" podUID="bf344860-07d2-4812-b993-d666d5669910" containerName="payload-processing" containerID="cri-o://2faf5e8f90b381075b6acbc2f910d94a0b864a5c4aa18a5b4cefeb6dab04e7ca" gracePeriod=25 Nov 02 01:31:12.501141 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:12.501083 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-6c45ad5e-575b6cb647-6nqrk" podUID="31acc518-a706-4c97-ac07-1201d7f58d4d" containerName="payload-pre-processing" containerID="cri-o://41feff1dbbffcdad1cfa0e796e956b1129a9b3f801ffd3f1d5951c7c25ede75f" gracePeriod=25 Nov 02 01:31:13.324033 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:13.323309 4427 generic.go:356] "Generic (PLEG): container finished" podID="bf344860-07d2-4812-b993-d666d5669910" containerID="2faf5e8f90b381075b6acbc2f910d94a0b864a5c4aa18a5b4cefeb6dab04e7ca" exitCode=0 Nov 02 01:31:13.324033 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:13.323529 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-6c45ad5e-5777dc9557-k6qq2" event={"ID":"bf344860-07d2-4812-b993-d666d5669910","Type":"ContainerDied","Data":"2faf5e8f90b381075b6acbc2f910d94a0b864a5c4aa18a5b4cefeb6dab04e7ca"} Nov 02 01:31:13.341849 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:13.333977 4427 generic.go:356] "Generic (PLEG): container finished" podID="31acc518-a706-4c97-ac07-1201d7f58d4d" containerID="41feff1dbbffcdad1cfa0e796e956b1129a9b3f801ffd3f1d5951c7c25ede75f" exitCode=0 Nov 02 01:31:13.341849 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:13.335131 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-6c45ad5e-575b6cb647-6nqrk" event={"ID":"31acc518-a706-4c97-ac07-1201d7f58d4d","Type":"ContainerDied","Data":"41feff1dbbffcdad1cfa0e796e956b1129a9b3f801ffd3f1d5951c7c25ede75f"} Nov 02 01:31:13.380983 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:13.380424 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-6c45ad5e-5777dc9557-k6qq2" Nov 02 01:31:13.383088 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:13.382954 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-6c45ad5e-575b6cb647-6nqrk" Nov 02 01:31:13.450402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:13.450260 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9a278165-5848cdd479-vq67l" podStartSLOduration=3.450224184 podStartE2EDuration="3.450224184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:31:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:31:12.297550165 +0000 UTC m=+976.918555392" watchObservedRunningTime="2026-11-02 01:31:13.450224184 +0000 UTC m=+978.071229396" Nov 02 01:31:13.510802 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:13.510011 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf344860-07d2-4812-b993-d666d5669910-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf344860-07d2-4812-b993-d666d5669910-plugins-config-volume\") pod \"bf344860-07d2-4812-b993-d666d5669910\" (UID: \"bf344860-07d2-4812-b993-d666d5669910\") " Nov 02 01:31:13.510802 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:13.510145 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31acc518-a706-4c97-ac07-1201d7f58d4d-kube-api-access-vndbw\" (UniqueName: \"kubernetes.io/projected/31acc518-a706-4c97-ac07-1201d7f58d4d-kube-api-access-vndbw\") pod \"31acc518-a706-4c97-ac07-1201d7f58d4d\" (UID: \"31acc518-a706-4c97-ac07-1201d7f58d4d\") " Nov 02 01:31:13.510802 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:13.510205 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf344860-07d2-4812-b993-d666d5669910-kube-api-access-rx8jw\" (UniqueName: \"kubernetes.io/projected/bf344860-07d2-4812-b993-d666d5669910-kube-api-access-rx8jw\") pod \"bf344860-07d2-4812-b993-d666d5669910\" (UID: \"bf344860-07d2-4812-b993-d666d5669910\") " Nov 02 01:31:13.510802 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:13.510320 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31acc518-a706-4c97-ac07-1201d7f58d4d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31acc518-a706-4c97-ac07-1201d7f58d4d-plugins-config-volume\") pod \"31acc518-a706-4c97-ac07-1201d7f58d4d\" (UID: \"31acc518-a706-4c97-ac07-1201d7f58d4d\") " Nov 02 01:31:13.515399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:13.512303 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf344860-07d2-4812-b993-d666d5669910-plugins-config-volume" pod "bf344860-07d2-4812-b993-d666d5669910" (UID: "bf344860-07d2-4812-b993-d666d5669910"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:31:13.515399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:13.512750 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31acc518-a706-4c97-ac07-1201d7f58d4d-plugins-config-volume" pod "31acc518-a706-4c97-ac07-1201d7f58d4d" (UID: "31acc518-a706-4c97-ac07-1201d7f58d4d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:31:13.627403 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:13.615557 4427 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf344860-07d2-4812-b993-d666d5669910-plugins-config-volume\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:31:13.627403 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:13.615595 4427 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31acc518-a706-4c97-ac07-1201d7f58d4d-plugins-config-volume\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:31:13.701673 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:13.701628 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf344860-07d2-4812-b993-d666d5669910-kube-api-access-rx8jw" pod "bf344860-07d2-4812-b993-d666d5669910" (UID: "bf344860-07d2-4812-b993-d666d5669910"). InnerVolumeSpecName "kube-api-access-rx8jw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:31:13.719889 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:13.719856 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rx8jw\" (UniqueName: \"kubernetes.io/projected/bf344860-07d2-4812-b993-d666d5669910-kube-api-access-rx8jw\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:31:14.021527 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:14.019563 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31acc518-a706-4c97-ac07-1201d7f58d4d-kube-api-access-vndbw" pod "31acc518-a706-4c97-ac07-1201d7f58d4d" (UID: "31acc518-a706-4c97-ac07-1201d7f58d4d"). InnerVolumeSpecName "kube-api-access-vndbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:31:14.128453 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:14.095237 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vndbw\" (UniqueName: \"kubernetes.io/projected/31acc518-a706-4c97-ac07-1201d7f58d4d-kube-api-access-vndbw\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:31:14.159488 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:14.158892 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-89833f31-b588945d7-s7d5t"] Nov 02 01:31:14.165908 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:14.160105 4427 cpu_manager.go:414] "containerMap: removing container" podUID="bf344860-07d2-4812-b993-d666d5669910" containerName="payload-processing" Nov 02 01:31:14.165908 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:14.160133 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf344860-07d2-4812-b993-d666d5669910" containerName="payload-processing" Nov 02 01:31:14.165908 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:14.160185 4427 cpu_manager.go:414] "containerMap: removing container" podUID="31acc518-a706-4c97-ac07-1201d7f58d4d" containerName="payload-pre-processing" Nov 02 01:31:14.165908 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:14.160195 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31acc518-a706-4c97-ac07-1201d7f58d4d" containerName="payload-pre-processing" Nov 02 01:31:14.165908 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:14.160350 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf344860-07d2-4812-b993-d666d5669910" containerName="payload-processing" Nov 02 01:31:14.165908 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:14.160368 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="31acc518-a706-4c97-ac07-1201d7f58d4d" containerName="payload-pre-processing" Nov 02 01:31:14.173600 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:14.168637 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-89833f31-b588945d7-s7d5t" Nov 02 01:31:14.180696 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:14.179223 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-89833f31\"" Nov 02 01:31:14.180696 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:14.179586 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-89833f31-dockercfg-fpgs9\"" Nov 02 01:31:14.213411 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:14.206796 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-89833f31-84fc7c596f-wrggk"] Nov 02 01:31:14.229620 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:14.225638 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-89833f31-84fc7c596f-wrggk" Nov 02 01:31:14.302449 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:14.301866 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f59ee4e-e9d2-4756-9d66-ff1b23ac6d34-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-89833f31-b588945d7-s7d5t\" (UID: \"8f59ee4e-e9d2-4756-9d66-ff1b23ac6d34\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-89833f31-b588945d7-s7d5t" Nov 02 01:31:14.302449 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:14.301910 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmbdt\" (UniqueName: \"kubernetes.io/projected/8f59ee4e-e9d2-4756-9d66-ff1b23ac6d34-kube-api-access-xmbdt\") pod \"payload-pre-processing-e2e-ait-89833f31-b588945d7-s7d5t\" (UID: \"8f59ee4e-e9d2-4756-9d66-ff1b23ac6d34\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-89833f31-b588945d7-s7d5t" Nov 02 01:31:14.429963 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:14.418939 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed8b7feb-9706-4bf5-9423-be3d0fa90604-plugins-config-volume\") pod \"payload-processing-e2e-ait-89833f31-84fc7c596f-wrggk\" (UID: \"ed8b7feb-9706-4bf5-9423-be3d0fa90604\") " pod="openshift-ingress/payload-processing-e2e-ait-89833f31-84fc7c596f-wrggk" Nov 02 01:31:14.429963 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:14.419015 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f59ee4e-e9d2-4756-9d66-ff1b23ac6d34-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-89833f31-b588945d7-s7d5t\" (UID: \"8f59ee4e-e9d2-4756-9d66-ff1b23ac6d34\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-89833f31-b588945d7-s7d5t" Nov 02 01:31:14.429963 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:14.419045 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmbdt\" (UniqueName: \"kubernetes.io/projected/8f59ee4e-e9d2-4756-9d66-ff1b23ac6d34-kube-api-access-xmbdt\") pod \"payload-pre-processing-e2e-ait-89833f31-b588945d7-s7d5t\" (UID: \"8f59ee4e-e9d2-4756-9d66-ff1b23ac6d34\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-89833f31-b588945d7-s7d5t" Nov 02 01:31:14.429963 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:14.419157 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwrfk\" (UniqueName: \"kubernetes.io/projected/ed8b7feb-9706-4bf5-9423-be3d0fa90604-kube-api-access-gwrfk\") pod \"payload-processing-e2e-ait-89833f31-84fc7c596f-wrggk\" (UID: \"ed8b7feb-9706-4bf5-9423-be3d0fa90604\") " pod="openshift-ingress/payload-processing-e2e-ait-89833f31-84fc7c596f-wrggk" Nov 02 01:31:14.429963 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:14.420292 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f59ee4e-e9d2-4756-9d66-ff1b23ac6d34-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-89833f31-b588945d7-s7d5t\" (UID: \"8f59ee4e-e9d2-4756-9d66-ff1b23ac6d34\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-89833f31-b588945d7-s7d5t" Nov 02 01:31:14.438563 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:14.437890 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-89833f31-b588945d7-s7d5t"] Nov 02 01:31:14.561578 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:14.530212 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed8b7feb-9706-4bf5-9423-be3d0fa90604-plugins-config-volume\") pod \"payload-processing-e2e-ait-89833f31-84fc7c596f-wrggk\" (UID: \"ed8b7feb-9706-4bf5-9423-be3d0fa90604\") " pod="openshift-ingress/payload-processing-e2e-ait-89833f31-84fc7c596f-wrggk" Nov 02 01:31:14.561578 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:14.530329 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwrfk\" (UniqueName: \"kubernetes.io/projected/ed8b7feb-9706-4bf5-9423-be3d0fa90604-kube-api-access-gwrfk\") pod \"payload-processing-e2e-ait-89833f31-84fc7c596f-wrggk\" (UID: \"ed8b7feb-9706-4bf5-9423-be3d0fa90604\") " pod="openshift-ingress/payload-processing-e2e-ait-89833f31-84fc7c596f-wrggk" Nov 02 01:31:14.561578 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:14.531873 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed8b7feb-9706-4bf5-9423-be3d0fa90604-plugins-config-volume\") pod \"payload-processing-e2e-ait-89833f31-84fc7c596f-wrggk\" (UID: \"ed8b7feb-9706-4bf5-9423-be3d0fa90604\") " pod="openshift-ingress/payload-processing-e2e-ait-89833f31-84fc7c596f-wrggk" Nov 02 01:31:14.561578 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:14.551180 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmbdt\" (UniqueName: \"kubernetes.io/projected/8f59ee4e-e9d2-4756-9d66-ff1b23ac6d34-kube-api-access-xmbdt\") pod \"payload-pre-processing-e2e-ait-89833f31-b588945d7-s7d5t\" (UID: \"8f59ee4e-e9d2-4756-9d66-ff1b23ac6d34\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-89833f31-b588945d7-s7d5t" Nov 02 01:31:14.606434 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:14.602909 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-89833f31-b588945d7-s7d5t" Nov 02 01:31:14.606434 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:14.605622 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-89833f31-84fc7c596f-wrggk"] Nov 02 01:31:14.664409 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:14.648023 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwrfk\" (UniqueName: \"kubernetes.io/projected/ed8b7feb-9706-4bf5-9423-be3d0fa90604-kube-api-access-gwrfk\") pod \"payload-processing-e2e-ait-89833f31-84fc7c596f-wrggk\" (UID: \"ed8b7feb-9706-4bf5-9423-be3d0fa90604\") " pod="openshift-ingress/payload-processing-e2e-ait-89833f31-84fc7c596f-wrggk" Nov 02 01:31:14.758511 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:14.758469 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-89833f31-84fc7c596f-wrggk" Nov 02 01:31:14.909401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:14.890907 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-6c45ad5e-5777dc9557-k6qq2" event={"ID":"bf344860-07d2-4812-b993-d666d5669910","Type":"ContainerDied","Data":"be5f356852de029724f8186f606e3a9a1112c7a78f129b0f612131f5435514ef"} Nov 02 01:31:14.909401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:14.890980 4427 scope.go:122] "RemoveContainer" containerID="2faf5e8f90b381075b6acbc2f910d94a0b864a5c4aa18a5b4cefeb6dab04e7ca" Nov 02 01:31:14.909401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:14.891137 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-6c45ad5e-5777dc9557-k6qq2" Nov 02 01:31:15.099428 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:15.093464 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-6c45ad5e-5777dc9557-k6qq2"] Nov 02 01:31:15.254434 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:15.252692 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-6c45ad5e-575b6cb647-6nqrk" Nov 02 01:31:15.257273 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:15.257236 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-6c45ad5e-575b6cb647-6nqrk" event={"ID":"31acc518-a706-4c97-ac07-1201d7f58d4d","Type":"ContainerDied","Data":"521bf0d549fe3447324c04726b9ffe0d2e0eeb0589da2e52ba8be4edee607c9b"} Nov 02 01:31:15.270870 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:15.270831 4427 scope.go:122] "RemoveContainer" containerID="41feff1dbbffcdad1cfa0e796e956b1129a9b3f801ffd3f1d5951c7c25ede75f" Nov 02 01:31:15.348835 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:15.348732 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-6c45ad5e-5777dc9557-k6qq2"] Nov 02 01:31:15.446422 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:15.445701 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-6c45ad5e-575b6cb647-6nqrk"] Nov 02 01:31:15.448403 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:15.448354 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-6c45ad5e-575b6cb647-6nqrk"] Nov 02 01:31:15.546349 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:15.546312 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31acc518-a706-4c97-ac07-1201d7f58d4d" path="/var/lib/kubelet/pods/31acc518-a706-4c97-ac07-1201d7f58d4d/volumes" Nov 02 01:31:15.547333 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:15.547312 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf344860-07d2-4812-b993-d666d5669910" path="/var/lib/kubelet/pods/bf344860-07d2-4812-b993-d666d5669910/volumes" Nov 02 01:31:15.608196 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:15.608157 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-89833f31-b588945d7-s7d5t"] Nov 02 01:31:15.746411 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:15.744681 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-89833f31-84fc7c596f-wrggk"] Nov 02 01:31:16.268461 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:16.266282 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-89833f31-b588945d7-s7d5t" event={"ID":"8f59ee4e-e9d2-4756-9d66-ff1b23ac6d34","Type":"ContainerStarted","Data":"5145bf3d0ebec6fe3e5bf5dc90e3a19f6154cd807832c751b03a10b989d53611"} Nov 02 01:31:16.268461 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:16.266344 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-89833f31-b588945d7-s7d5t" event={"ID":"8f59ee4e-e9d2-4756-9d66-ff1b23ac6d34","Type":"ContainerStarted","Data":"0508c8811ff956e304bff7ccd2d2e30cb3184b4492d8324201caa5b8c3d9cc19"} Nov 02 01:31:16.268461 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:16.266639 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-89833f31-b588945d7-s7d5t" Nov 02 01:31:16.272952 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:16.272897 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-89833f31-84fc7c596f-wrggk" event={"ID":"ed8b7feb-9706-4bf5-9423-be3d0fa90604","Type":"ContainerStarted","Data":"62eb481823fb869a8d6b340087ce6a14db67e50ea91a6f5c9a5873d37fd575bd"} Nov 02 01:31:16.272952 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:16.272955 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-89833f31-84fc7c596f-wrggk" event={"ID":"ed8b7feb-9706-4bf5-9423-be3d0fa90604","Type":"ContainerStarted","Data":"f9092bacb2f1938d42efd8f16571e213afcfbac40b8706aee4338a0916e2ab08"} Nov 02 01:31:16.273226 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:16.273081 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-89833f31-84fc7c596f-wrggk" Nov 02 01:31:16.289092 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:16.289044 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-89833f31-b588945d7-s7d5t" podStartSLOduration=3.289031039 podStartE2EDuration="3.289031039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:31:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:31:16.285276541 +0000 UTC m=+980.906281753" watchObservedRunningTime="2026-11-02 01:31:16.289031039 +0000 UTC m=+980.910036249" Nov 02 01:31:16.309349 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:16.309289 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-89833f31-84fc7c596f-wrggk" podStartSLOduration=3.309268632 podStartE2EDuration="3.309268632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:31:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:31:16.30745225 +0000 UTC m=+980.928457463" watchObservedRunningTime="2026-11-02 01:31:16.309268632 +0000 UTC m=+980.930273845" Nov 02 01:31:25.256030 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:25.255994 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9a278165-5848cdd479-vq67l" Nov 02 01:31:27.284631 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:27.284600 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-89833f31-b588945d7-s7d5t" Nov 02 01:31:27.285104 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:27.284997 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-89833f31-84fc7c596f-wrggk" Nov 02 01:31:31.651896 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:31:31.651658 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:31:38.091401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:38.039698 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" podUID="366c78ee-4aaf-4b7c-84b4-17028538cb1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 01:31:38.093443 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:38.047459 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" podUID="47a9069f-1ffa-4c83-be8b-5e62f1d3275a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:31:38.126759 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:38.126467 4427 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" podUID="fb533b5f-3525-454a-8ce2-ea9246194f28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:31:38.327148 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:38.303139 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" podUID="ea61971c-5a72-4609-b5a0-58a74b324bb7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:31:38.328731 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:38.328694 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" podUID="f67c954a-ce5b-41d9-8d99-7d126bd89d59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:31:38.362581 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:38.362543 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" podUID="7085c0ab-7ee3-4665-abad-2977c25f68ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 01:31:38.442853 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:38.442805 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" podUID="353a8a41-21ee-4470-861c-b53833beb106" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:31:38.475413 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:38.451111 4427 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" podUID="c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": context deadline exceeded" Nov 02 01:31:38.554914 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:38.554833 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" podUID="d73ae484-a9b8-4565-a346-0d803a1981ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:31:38.651196 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:38.651092 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" podUID="c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:31:38.723399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:38.701548 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" podUID="d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:31:38.749621 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:38.749579 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" podUID="706df1bd-19cf-4ab9-9aaa-3c12c97ccf45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:31:39.014406 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:39.014273 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" podUID="eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:31:39.025212 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:39.023875 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" podUID="cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:31:45.731461 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:45.731424 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9a278165-5848cdd479-vq67l"] Nov 02 01:31:45.957538 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:45.957501 4427 generic.go:356] "Generic (PLEG): container finished" podID="a221bfd5-295a-456b-ae11-afe35719f55d" containerID="da570a30e484ce248c307d15721accd19ef19b786d460e609d87d9272878911d" exitCode=0 Nov 02 01:31:45.957715 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:45.957556 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww" event={"ID":"a221bfd5-295a-456b-ae11-afe35719f55d","Type":"ContainerDied","Data":"da570a30e484ce248c307d15721accd19ef19b786d460e609d87d9272878911d"} Nov 02 01:31:45.958080 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:45.958064 4427 scope.go:122] "RemoveContainer" containerID="da570a30e484ce248c307d15721accd19ef19b786d460e609d87d9272878911d" Nov 02 01:31:45.959842 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:45.959813 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:31:45.959919 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:45.959891 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:31:45.959955 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:45.959923 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:31:46.992580 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:46.992539 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww" event={"ID":"a221bfd5-295a-456b-ae11-afe35719f55d","Type":"ContainerStarted","Data":"1e74144f04438487cf3698044906279c5fb08f213d74d46227fb366369f3aa30"} Nov 02 01:31:47.645498 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:47.645456 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww" Nov 02 01:31:47.645679 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:47.645521 4427 patch_prober.go:29] interesting pod/e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 02 01:31:47.645679 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:47.645575 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww" podUID="a221bfd5-295a-456b-ae11-afe35719f55d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 02 01:31:48.033096 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:48.033006 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-89833f31-84fc7c596f-wrggk"] Nov 02 01:31:48.047973 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:48.047939 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-89833f31-b588945d7-s7d5t"] Nov 02 01:31:48.645048 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:48.645005 4427 patch_prober.go:29] interesting pod/e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 02 01:31:48.645250 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:48.645074 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww" podUID="a221bfd5-295a-456b-ae11-afe35719f55d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 02 01:31:49.456634 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:49.447748 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww"] Nov 02 01:31:49.456634 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:49.448019 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww" podUID="a221bfd5-295a-456b-ae11-afe35719f55d" containerName="istio-proxy" containerID="cri-o://1e74144f04438487cf3698044906279c5fb08f213d74d46227fb366369f3aa30" gracePeriod=30 Nov 02 01:31:50.734488 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:50.734443 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9a278165-5848cdd479-vq67l" podUID="4e04e863-2c20-4c9e-91e2-2c6ecf4bb627" containerName="payload-pre-processing" containerID="cri-o://437bc0cea8cb431dd25836fe9aa303872722bbbd43fd1d2bc7a7ec3e2ae35126" gracePeriod=25 Nov 02 01:31:51.075107 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:51.075052 4427 generic.go:356] "Generic (PLEG): container finished" podID="4e04e863-2c20-4c9e-91e2-2c6ecf4bb627" containerID="437bc0cea8cb431dd25836fe9aa303872722bbbd43fd1d2bc7a7ec3e2ae35126" exitCode=0 Nov 02 01:31:51.075506 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:51.075482 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9a278165-5848cdd479-vq67l" event={"ID":"4e04e863-2c20-4c9e-91e2-2c6ecf4bb627","Type":"ContainerDied","Data":"437bc0cea8cb431dd25836fe9aa303872722bbbd43fd1d2bc7a7ec3e2ae35126"} Nov 02 01:31:51.304550 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:51.304525 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9a278165-5848cdd479-vq67l" Nov 02 01:31:51.330450 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:51.329564 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e04e863-2c20-4c9e-91e2-2c6ecf4bb627-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e04e863-2c20-4c9e-91e2-2c6ecf4bb627-plugins-config-volume\") pod \"4e04e863-2c20-4c9e-91e2-2c6ecf4bb627\" (UID: \"4e04e863-2c20-4c9e-91e2-2c6ecf4bb627\") " Nov 02 01:31:51.330450 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:51.329759 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e04e863-2c20-4c9e-91e2-2c6ecf4bb627-kube-api-access-hmsbv\" (UniqueName: \"kubernetes.io/projected/4e04e863-2c20-4c9e-91e2-2c6ecf4bb627-kube-api-access-hmsbv\") pod \"4e04e863-2c20-4c9e-91e2-2c6ecf4bb627\" (UID: \"4e04e863-2c20-4c9e-91e2-2c6ecf4bb627\") " Nov 02 01:31:51.331560 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:51.331528 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e04e863-2c20-4c9e-91e2-2c6ecf4bb627-plugins-config-volume" pod "4e04e863-2c20-4c9e-91e2-2c6ecf4bb627" (UID: "4e04e863-2c20-4c9e-91e2-2c6ecf4bb627"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:31:51.337148 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:51.333998 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e04e863-2c20-4c9e-91e2-2c6ecf4bb627-kube-api-access-hmsbv" pod "4e04e863-2c20-4c9e-91e2-2c6ecf4bb627" (UID: "4e04e863-2c20-4c9e-91e2-2c6ecf4bb627"). InnerVolumeSpecName "kube-api-access-hmsbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:31:51.432289 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:51.432249 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmsbv\" (UniqueName: \"kubernetes.io/projected/4e04e863-2c20-4c9e-91e2-2c6ecf4bb627-kube-api-access-hmsbv\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:31:51.432289 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:51.432276 4427 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e04e863-2c20-4c9e-91e2-2c6ecf4bb627-plugins-config-volume\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:31:52.100588 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:52.100545 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9a278165-5848cdd479-vq67l" event={"ID":"4e04e863-2c20-4c9e-91e2-2c6ecf4bb627","Type":"ContainerDied","Data":"22fc8766d83d1a1e44f28f5f6dce14207b00179dcf85ea035912489aba994af8"} Nov 02 01:31:52.100588 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:52.100579 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9a278165-5848cdd479-vq67l" Nov 02 01:31:52.101093 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:52.100654 4427 scope.go:122] "RemoveContainer" containerID="437bc0cea8cb431dd25836fe9aa303872722bbbd43fd1d2bc7a7ec3e2ae35126" Nov 02 01:31:52.123357 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:52.123315 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9a278165-5848cdd479-vq67l"] Nov 02 01:31:52.127196 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:52.127171 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9a278165-5848cdd479-vq67l"] Nov 02 01:31:53.034402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:53.034194 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-89833f31-84fc7c596f-wrggk" podUID="ed8b7feb-9706-4bf5-9423-be3d0fa90604" containerName="payload-processing" containerID="cri-o://62eb481823fb869a8d6b340087ce6a14db67e50ea91a6f5c9a5873d37fd575bd" gracePeriod=25 Nov 02 01:31:53.051420 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:53.049568 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-89833f31-b588945d7-s7d5t" podUID="8f59ee4e-e9d2-4756-9d66-ff1b23ac6d34" containerName="payload-pre-processing" containerID="cri-o://5145bf3d0ebec6fe3e5bf5dc90e3a19f6154cd807832c751b03a10b989d53611" gracePeriod=25 Nov 02 01:31:53.535568 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:53.535138 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-89833f31-84fc7c596f-wrggk" Nov 02 01:31:53.539525 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:53.539489 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e04e863-2c20-4c9e-91e2-2c6ecf4bb627" path="/var/lib/kubelet/pods/4e04e863-2c20-4c9e-91e2-2c6ecf4bb627/volumes" Nov 02 01:31:53.540796 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:53.540773 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-89833f31-b588945d7-s7d5t" Nov 02 01:31:53.638495 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:53.638457 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f59ee4e-e9d2-4756-9d66-ff1b23ac6d34-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f59ee4e-e9d2-4756-9d66-ff1b23ac6d34-plugins-config-volume\") pod \"8f59ee4e-e9d2-4756-9d66-ff1b23ac6d34\" (UID: \"8f59ee4e-e9d2-4756-9d66-ff1b23ac6d34\") " Nov 02 01:31:53.638739 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:53.638557 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed8b7feb-9706-4bf5-9423-be3d0fa90604-kube-api-access-gwrfk\" (UniqueName: \"kubernetes.io/projected/ed8b7feb-9706-4bf5-9423-be3d0fa90604-kube-api-access-gwrfk\") pod \"ed8b7feb-9706-4bf5-9423-be3d0fa90604\" (UID: \"ed8b7feb-9706-4bf5-9423-be3d0fa90604\") " Nov 02 01:31:53.638739 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:53.638602 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f59ee4e-e9d2-4756-9d66-ff1b23ac6d34-kube-api-access-xmbdt\" (UniqueName: \"kubernetes.io/projected/8f59ee4e-e9d2-4756-9d66-ff1b23ac6d34-kube-api-access-xmbdt\") pod \"8f59ee4e-e9d2-4756-9d66-ff1b23ac6d34\" (UID: \"8f59ee4e-e9d2-4756-9d66-ff1b23ac6d34\") " Nov 02 01:31:53.638739 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:53.638642 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed8b7feb-9706-4bf5-9423-be3d0fa90604-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed8b7feb-9706-4bf5-9423-be3d0fa90604-plugins-config-volume\") pod \"ed8b7feb-9706-4bf5-9423-be3d0fa90604\" (UID: \"ed8b7feb-9706-4bf5-9423-be3d0fa90604\") " Nov 02 01:31:53.638926 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:53.638902 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f59ee4e-e9d2-4756-9d66-ff1b23ac6d34-plugins-config-volume" pod "8f59ee4e-e9d2-4756-9d66-ff1b23ac6d34" (UID: "8f59ee4e-e9d2-4756-9d66-ff1b23ac6d34"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:31:53.639480 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:53.639459 4427 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f59ee4e-e9d2-4756-9d66-ff1b23ac6d34-plugins-config-volume\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:31:53.639480 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:53.639464 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed8b7feb-9706-4bf5-9423-be3d0fa90604-plugins-config-volume" pod "ed8b7feb-9706-4bf5-9423-be3d0fa90604" (UID: "ed8b7feb-9706-4bf5-9423-be3d0fa90604"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:31:53.641759 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:53.641733 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f59ee4e-e9d2-4756-9d66-ff1b23ac6d34-kube-api-access-xmbdt" pod "8f59ee4e-e9d2-4756-9d66-ff1b23ac6d34" (UID: "8f59ee4e-e9d2-4756-9d66-ff1b23ac6d34"). InnerVolumeSpecName "kube-api-access-xmbdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:31:53.641846 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:53.641769 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed8b7feb-9706-4bf5-9423-be3d0fa90604-kube-api-access-gwrfk" pod "ed8b7feb-9706-4bf5-9423-be3d0fa90604" (UID: "ed8b7feb-9706-4bf5-9423-be3d0fa90604"). InnerVolumeSpecName "kube-api-access-gwrfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:31:53.741511 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:53.741480 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwrfk\" (UniqueName: \"kubernetes.io/projected/ed8b7feb-9706-4bf5-9423-be3d0fa90604-kube-api-access-gwrfk\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:31:53.741511 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:53.741509 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmbdt\" (UniqueName: \"kubernetes.io/projected/8f59ee4e-e9d2-4756-9d66-ff1b23ac6d34-kube-api-access-xmbdt\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:31:53.741511 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:53.741519 4427 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed8b7feb-9706-4bf5-9423-be3d0fa90604-plugins-config-volume\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:31:54.063637 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:54.063520 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww" Nov 02 01:31:54.134264 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:54.134225 4427 generic.go:356] "Generic (PLEG): container finished" podID="8f59ee4e-e9d2-4756-9d66-ff1b23ac6d34" containerID="5145bf3d0ebec6fe3e5bf5dc90e3a19f6154cd807832c751b03a10b989d53611" exitCode=0 Nov 02 01:31:54.134557 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:54.134530 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-89833f31-b588945d7-s7d5t" event={"ID":"8f59ee4e-e9d2-4756-9d66-ff1b23ac6d34","Type":"ContainerDied","Data":"5145bf3d0ebec6fe3e5bf5dc90e3a19f6154cd807832c751b03a10b989d53611"} Nov 02 01:31:54.134623 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:54.134607 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-89833f31-b588945d7-s7d5t" event={"ID":"8f59ee4e-e9d2-4756-9d66-ff1b23ac6d34","Type":"ContainerDied","Data":"0508c8811ff956e304bff7ccd2d2e30cb3184b4492d8324201caa5b8c3d9cc19"} Nov 02 01:31:54.134673 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:54.134638 4427 scope.go:122] "RemoveContainer" containerID="5145bf3d0ebec6fe3e5bf5dc90e3a19f6154cd807832c751b03a10b989d53611" Nov 02 01:31:54.134772 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:54.134758 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-89833f31-b588945d7-s7d5t" Nov 02 01:31:54.140615 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:54.140585 4427 generic.go:356] "Generic (PLEG): container finished" podID="ed8b7feb-9706-4bf5-9423-be3d0fa90604" containerID="62eb481823fb869a8d6b340087ce6a14db67e50ea91a6f5c9a5873d37fd575bd" exitCode=0 Nov 02 01:31:54.140750 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:54.140733 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-89833f31-84fc7c596f-wrggk" event={"ID":"ed8b7feb-9706-4bf5-9423-be3d0fa90604","Type":"ContainerDied","Data":"62eb481823fb869a8d6b340087ce6a14db67e50ea91a6f5c9a5873d37fd575bd"} Nov 02 01:31:54.140824 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:54.140786 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-89833f31-84fc7c596f-wrggk" event={"ID":"ed8b7feb-9706-4bf5-9423-be3d0fa90604","Type":"ContainerDied","Data":"f9092bacb2f1938d42efd8f16571e213afcfbac40b8706aee4338a0916e2ab08"} Nov 02 01:31:54.140883 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:54.140861 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-89833f31-84fc7c596f-wrggk" Nov 02 01:31:54.150041 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:54.150019 4427 scope.go:122] "RemoveContainer" containerID="5145bf3d0ebec6fe3e5bf5dc90e3a19f6154cd807832c751b03a10b989d53611" Nov 02 01:31:54.150462 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:31:54.150434 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5145bf3d0ebec6fe3e5bf5dc90e3a19f6154cd807832c751b03a10b989d53611\": container with ID starting with 5145bf3d0ebec6fe3e5bf5dc90e3a19f6154cd807832c751b03a10b989d53611 not found: ID does not exist" containerID="5145bf3d0ebec6fe3e5bf5dc90e3a19f6154cd807832c751b03a10b989d53611" Nov 02 01:31:54.150599 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:54.150475 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5145bf3d0ebec6fe3e5bf5dc90e3a19f6154cd807832c751b03a10b989d53611"} err="failed to get container status \"5145bf3d0ebec6fe3e5bf5dc90e3a19f6154cd807832c751b03a10b989d53611\": rpc error: code = NotFound desc = could not find container \"5145bf3d0ebec6fe3e5bf5dc90e3a19f6154cd807832c751b03a10b989d53611\": container with ID starting with 5145bf3d0ebec6fe3e5bf5dc90e3a19f6154cd807832c751b03a10b989d53611 not found: ID does not exist" Nov 02 01:31:54.150599 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:54.150498 4427 scope.go:122] "RemoveContainer" containerID="62eb481823fb869a8d6b340087ce6a14db67e50ea91a6f5c9a5873d37fd575bd" Nov 02 01:31:54.165030 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:54.164927 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-89833f31-b588945d7-s7d5t"] Nov 02 01:31:54.168200 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:54.168174 4427 scope.go:122] "RemoveContainer" containerID="62eb481823fb869a8d6b340087ce6a14db67e50ea91a6f5c9a5873d37fd575bd" Nov 02 01:31:54.170482 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:31:54.170447 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62eb481823fb869a8d6b340087ce6a14db67e50ea91a6f5c9a5873d37fd575bd\": container with ID starting with 62eb481823fb869a8d6b340087ce6a14db67e50ea91a6f5c9a5873d37fd575bd not found: ID does not exist" containerID="62eb481823fb869a8d6b340087ce6a14db67e50ea91a6f5c9a5873d37fd575bd" Nov 02 01:31:54.170612 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:54.170491 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62eb481823fb869a8d6b340087ce6a14db67e50ea91a6f5c9a5873d37fd575bd"} err="failed to get container status \"62eb481823fb869a8d6b340087ce6a14db67e50ea91a6f5c9a5873d37fd575bd\": rpc error: code = NotFound desc = could not find container \"62eb481823fb869a8d6b340087ce6a14db67e50ea91a6f5c9a5873d37fd575bd\": container with ID starting with 62eb481823fb869a8d6b340087ce6a14db67e50ea91a6f5c9a5873d37fd575bd not found: ID does not exist" Nov 02 01:31:54.171943 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:54.171921 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-89833f31-b588945d7-s7d5t"] Nov 02 01:31:54.185996 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:54.185957 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-89833f31-84fc7c596f-wrggk"] Nov 02 01:31:54.192051 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:54.192025 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-89833f31-84fc7c596f-wrggk"] Nov 02 01:31:55.536444 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:55.536410 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f59ee4e-e9d2-4756-9d66-ff1b23ac6d34" path="/var/lib/kubelet/pods/8f59ee4e-e9d2-4756-9d66-ff1b23ac6d34/volumes" Nov 02 01:31:55.537000 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:55.536979 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed8b7feb-9706-4bf5-9423-be3d0fa90604" path="/var/lib/kubelet/pods/ed8b7feb-9706-4bf5-9423-be3d0fa90604/volumes" Nov 02 01:31:57.940058 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:31:57.940019 4427 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda221bfd5_295a_456b_ae11_afe35719f55d.slice/crio-1e74144f04438487cf3698044906279c5fb08f213d74d46227fb366369f3aa30.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda221bfd5_295a_456b_ae11_afe35719f55d.slice/crio-7cbf2069de9ed8a550fe9ffc697559776ee386b8cff4eef8efd1aeaf8ab07bd7\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda221bfd5_295a_456b_ae11_afe35719f55d.slice/crio-conmon-1e74144f04438487cf3698044906279c5fb08f213d74d46227fb366369f3aa30.scope\": RecentStats: unable to find data in memory cache]" Nov 02 01:31:58.280834 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:58.276355 4427 generic.go:356] "Generic (PLEG): container finished" podID="a221bfd5-295a-456b-ae11-afe35719f55d" containerID="1e74144f04438487cf3698044906279c5fb08f213d74d46227fb366369f3aa30" exitCode=0 Nov 02 01:31:58.280834 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:58.276526 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww" event={"ID":"a221bfd5-295a-456b-ae11-afe35719f55d","Type":"ContainerDied","Data":"1e74144f04438487cf3698044906279c5fb08f213d74d46227fb366369f3aa30"} Nov 02 01:31:58.280834 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:58.276595 4427 scope.go:122] "RemoveContainer" containerID="da570a30e484ce248c307d15721accd19ef19b786d460e609d87d9272878911d" Nov 02 01:31:58.411105 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:58.411067 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww" Nov 02 01:31:58.500114 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:58.500075 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a221bfd5-295a-456b-ae11-afe35719f55d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a221bfd5-295a-456b-ae11-afe35719f55d-istio-envoy\") pod \"a221bfd5-295a-456b-ae11-afe35719f55d\" (UID: \"a221bfd5-295a-456b-ae11-afe35719f55d\") " Nov 02 01:31:58.500312 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:58.500159 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a221bfd5-295a-456b-ae11-afe35719f55d-kube-api-access-dldlv\" (UniqueName: \"kubernetes.io/projected/a221bfd5-295a-456b-ae11-afe35719f55d-kube-api-access-dldlv\") pod \"a221bfd5-295a-456b-ae11-afe35719f55d\" (UID: \"a221bfd5-295a-456b-ae11-afe35719f55d\") " Nov 02 01:31:58.500312 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:58.500247 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a221bfd5-295a-456b-ae11-afe35719f55d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a221bfd5-295a-456b-ae11-afe35719f55d-credential-socket\") pod \"a221bfd5-295a-456b-ae11-afe35719f55d\" (UID: \"a221bfd5-295a-456b-ae11-afe35719f55d\") " Nov 02 01:31:58.500501 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:58.500337 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a221bfd5-295a-456b-ae11-afe35719f55d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a221bfd5-295a-456b-ae11-afe35719f55d-istio-data\") pod \"a221bfd5-295a-456b-ae11-afe35719f55d\" (UID: \"a221bfd5-295a-456b-ae11-afe35719f55d\") " Nov 02 01:31:58.500501 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:58.500414 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a221bfd5-295a-456b-ae11-afe35719f55d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a221bfd5-295a-456b-ae11-afe35719f55d-workload-certs\") pod \"a221bfd5-295a-456b-ae11-afe35719f55d\" (UID: \"a221bfd5-295a-456b-ae11-afe35719f55d\") " Nov 02 01:31:58.500714 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:58.500642 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a221bfd5-295a-456b-ae11-afe35719f55d-credential-socket" pod "a221bfd5-295a-456b-ae11-afe35719f55d" (UID: "a221bfd5-295a-456b-ae11-afe35719f55d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:31:58.500714 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:58.500635 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a221bfd5-295a-456b-ae11-afe35719f55d-istio-data" pod "a221bfd5-295a-456b-ae11-afe35719f55d" (UID: "a221bfd5-295a-456b-ae11-afe35719f55d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:31:58.500847 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:58.500728 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a221bfd5-295a-456b-ae11-afe35719f55d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a221bfd5-295a-456b-ae11-afe35719f55d-workload-socket\") pod \"a221bfd5-295a-456b-ae11-afe35719f55d\" (UID: \"a221bfd5-295a-456b-ae11-afe35719f55d\") " Nov 02 01:31:58.500847 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:58.500770 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a221bfd5-295a-456b-ae11-afe35719f55d-istio-token\" (UniqueName: \"kubernetes.io/projected/a221bfd5-295a-456b-ae11-afe35719f55d-istio-token\") pod \"a221bfd5-295a-456b-ae11-afe35719f55d\" (UID: \"a221bfd5-295a-456b-ae11-afe35719f55d\") " Nov 02 01:31:58.500847 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:58.500808 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a221bfd5-295a-456b-ae11-afe35719f55d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a221bfd5-295a-456b-ae11-afe35719f55d-istio-podinfo\") pod \"a221bfd5-295a-456b-ae11-afe35719f55d\" (UID: \"a221bfd5-295a-456b-ae11-afe35719f55d\") " Nov 02 01:31:58.501170 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:58.500848 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a221bfd5-295a-456b-ae11-afe35719f55d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a221bfd5-295a-456b-ae11-afe35719f55d-istiod-ca-cert\") pod \"a221bfd5-295a-456b-ae11-afe35719f55d\" (UID: \"a221bfd5-295a-456b-ae11-afe35719f55d\") " Nov 02 01:31:58.501170 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:58.500957 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a221bfd5-295a-456b-ae11-afe35719f55d-workload-certs" pod "a221bfd5-295a-456b-ae11-afe35719f55d" (UID: "a221bfd5-295a-456b-ae11-afe35719f55d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:31:58.501782 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:58.501663 4427 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a221bfd5-295a-456b-ae11-afe35719f55d-credential-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:31:58.501782 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:58.501681 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a221bfd5-295a-456b-ae11-afe35719f55d-istiod-ca-cert" pod "a221bfd5-295a-456b-ae11-afe35719f55d" (UID: "a221bfd5-295a-456b-ae11-afe35719f55d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:31:58.501782 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:58.501691 4427 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a221bfd5-295a-456b-ae11-afe35719f55d-istio-data\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:31:58.501782 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:58.501729 4427 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a221bfd5-295a-456b-ae11-afe35719f55d-workload-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:31:58.502140 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:58.502063 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a221bfd5-295a-456b-ae11-afe35719f55d-workload-socket" pod "a221bfd5-295a-456b-ae11-afe35719f55d" (UID: "a221bfd5-295a-456b-ae11-afe35719f55d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:31:58.502911 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:58.502877 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a221bfd5-295a-456b-ae11-afe35719f55d-istio-envoy" pod "a221bfd5-295a-456b-ae11-afe35719f55d" (UID: "a221bfd5-295a-456b-ae11-afe35719f55d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:31:58.503033 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:58.503011 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a221bfd5-295a-456b-ae11-afe35719f55d-kube-api-access-dldlv" pod "a221bfd5-295a-456b-ae11-afe35719f55d" (UID: "a221bfd5-295a-456b-ae11-afe35719f55d"). InnerVolumeSpecName "kube-api-access-dldlv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:31:58.503336 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:58.503299 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a221bfd5-295a-456b-ae11-afe35719f55d-istio-token" pod "a221bfd5-295a-456b-ae11-afe35719f55d" (UID: "a221bfd5-295a-456b-ae11-afe35719f55d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:31:58.503472 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:58.503448 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a221bfd5-295a-456b-ae11-afe35719f55d-istio-podinfo" pod "a221bfd5-295a-456b-ae11-afe35719f55d" (UID: "a221bfd5-295a-456b-ae11-afe35719f55d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:31:58.603905 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:58.603872 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dldlv\" (UniqueName: \"kubernetes.io/projected/a221bfd5-295a-456b-ae11-afe35719f55d-kube-api-access-dldlv\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:31:58.603905 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:58.603900 4427 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a221bfd5-295a-456b-ae11-afe35719f55d-workload-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:31:58.603905 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:58.603910 4427 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a221bfd5-295a-456b-ae11-afe35719f55d-istio-token\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:31:58.604240 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:58.603999 4427 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a221bfd5-295a-456b-ae11-afe35719f55d-istio-podinfo\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:31:58.604240 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:58.604028 4427 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a221bfd5-295a-456b-ae11-afe35719f55d-istiod-ca-cert\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:31:58.604240 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:58.604044 4427 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a221bfd5-295a-456b-ae11-afe35719f55d-istio-envoy\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:31:59.284690 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:59.284659 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww" Nov 02 01:31:59.285209 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:59.284654 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww" event={"ID":"a221bfd5-295a-456b-ae11-afe35719f55d","Type":"ContainerDied","Data":"7cbf2069de9ed8a550fe9ffc697559776ee386b8cff4eef8efd1aeaf8ab07bd7"} Nov 02 01:31:59.285209 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:59.284783 4427 scope.go:122] "RemoveContainer" containerID="1e74144f04438487cf3698044906279c5fb08f213d74d46227fb366369f3aa30" Nov 02 01:31:59.312686 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:59.312652 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww"] Nov 02 01:31:59.319397 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:59.319343 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-9a278165-gw-openshift-default-55df654c97-vx6ww"] Nov 02 01:31:59.537278 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:31:59.537184 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a221bfd5-295a-456b-ae11-afe35719f55d" path="/var/lib/kubelet/pods/a221bfd5-295a-456b-ae11-afe35719f55d/volumes" Nov 02 01:32:01.584632 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:32:01.584591 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:32:05.029862 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:05.029828 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3190c013-k92s4"] Nov 02 01:32:05.033817 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:05.033791 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3190c013-k92s4"] Nov 02 01:32:05.545800 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:05.545767 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92b288ee-b9e7-494a-a48b-0452b09e9f03" path="/var/lib/kubelet/pods/92b288ee-b9e7-494a-a48b-0452b09e9f03/volumes" Nov 02 01:32:12.045455 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:12.045358 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96095420-4rp8x"] Nov 02 01:32:12.049728 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:12.049690 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96095420-4rp8x"] Nov 02 01:32:13.535081 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:13.535015 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67b125a9-f02b-416c-8108-e4a8a41f1001" path="/var/lib/kubelet/pods/67b125a9-f02b-416c-8108-e4a8a41f1001/volumes" Nov 02 01:32:17.797399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:17.792788 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d"] Nov 02 01:32:17.797399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:17.793775 4427 cpu_manager.go:414] "containerMap: removing container" podUID="a221bfd5-295a-456b-ae11-afe35719f55d" containerName="istio-proxy" Nov 02 01:32:17.797399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:17.793795 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a221bfd5-295a-456b-ae11-afe35719f55d" containerName="istio-proxy" Nov 02 01:32:17.797399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:17.793830 4427 cpu_manager.go:414] "containerMap: removing container" podUID="4e04e863-2c20-4c9e-91e2-2c6ecf4bb627" containerName="payload-pre-processing" Nov 02 01:32:17.797399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:17.793838 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e04e863-2c20-4c9e-91e2-2c6ecf4bb627" containerName="payload-pre-processing" Nov 02 01:32:17.797399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:17.793851 4427 cpu_manager.go:414] "containerMap: removing container" podUID="ed8b7feb-9706-4bf5-9423-be3d0fa90604" containerName="payload-processing" Nov 02 01:32:17.797399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:17.793857 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed8b7feb-9706-4bf5-9423-be3d0fa90604" containerName="payload-processing" Nov 02 01:32:17.797399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:17.793871 4427 cpu_manager.go:414] "containerMap: removing container" podUID="8f59ee4e-e9d2-4756-9d66-ff1b23ac6d34" containerName="payload-pre-processing" Nov 02 01:32:17.797399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:17.793877 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f59ee4e-e9d2-4756-9d66-ff1b23ac6d34" containerName="payload-pre-processing" Nov 02 01:32:17.800736 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:17.800705 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="a221bfd5-295a-456b-ae11-afe35719f55d" containerName="istio-proxy" Nov 02 01:32:17.800962 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:17.800948 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e04e863-2c20-4c9e-91e2-2c6ecf4bb627" containerName="payload-pre-processing" Nov 02 01:32:17.801075 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:17.801049 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed8b7feb-9706-4bf5-9423-be3d0fa90604" containerName="payload-processing" Nov 02 01:32:17.801160 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:17.801149 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f59ee4e-e9d2-4756-9d66-ff1b23ac6d34" containerName="payload-pre-processing" Nov 02 01:32:17.832913 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:17.832876 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d" Nov 02 01:32:17.838510 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:17.838479 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d"] Nov 02 01:32:17.853665 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:17.853630 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-c22a0d-openshift-default-dockercfg-pj86k\"" Nov 02 01:32:17.936219 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:17.936167 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6da871e-dedf-412b-bbad-4257087b71a7-istio-data\") pod \"e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d\" (UID: \"f6da871e-dedf-412b-bbad-4257087b71a7\") " pod="openshift-ingress/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d" Nov 02 01:32:17.938525 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:17.938488 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6da871e-dedf-412b-bbad-4257087b71a7-credential-socket\") pod \"e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d\" (UID: \"f6da871e-dedf-412b-bbad-4257087b71a7\") " pod="openshift-ingress/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d" Nov 02 01:32:17.938909 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:17.938878 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6da871e-dedf-412b-bbad-4257087b71a7-istio-token\") pod \"e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d\" (UID: \"f6da871e-dedf-412b-bbad-4257087b71a7\") " pod="openshift-ingress/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d" Nov 02 01:32:17.939080 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:17.939056 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6da871e-dedf-412b-bbad-4257087b71a7-istio-podinfo\") pod \"e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d\" (UID: \"f6da871e-dedf-412b-bbad-4257087b71a7\") " pod="openshift-ingress/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d" Nov 02 01:32:17.939236 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:17.939211 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6da871e-dedf-412b-bbad-4257087b71a7-workload-socket\") pod \"e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d\" (UID: \"f6da871e-dedf-412b-bbad-4257087b71a7\") " pod="openshift-ingress/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d" Nov 02 01:32:17.939582 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:17.939550 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6da871e-dedf-412b-bbad-4257087b71a7-istio-envoy\") pod \"e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d\" (UID: \"f6da871e-dedf-412b-bbad-4257087b71a7\") " pod="openshift-ingress/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d" Nov 02 01:32:17.939742 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:17.939726 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6da871e-dedf-412b-bbad-4257087b71a7-istiod-ca-cert\") pod \"e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d\" (UID: \"f6da871e-dedf-412b-bbad-4257087b71a7\") " pod="openshift-ingress/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d" Nov 02 01:32:17.939887 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:17.939870 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6da871e-dedf-412b-bbad-4257087b71a7-workload-certs\") pod \"e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d\" (UID: \"f6da871e-dedf-412b-bbad-4257087b71a7\") " pod="openshift-ingress/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d" Nov 02 01:32:17.950588 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:17.950556 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvf22\" (UniqueName: \"kubernetes.io/projected/f6da871e-dedf-412b-bbad-4257087b71a7-kube-api-access-dvf22\") pod \"e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d\" (UID: \"f6da871e-dedf-412b-bbad-4257087b71a7\") " pod="openshift-ingress/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d" Nov 02 01:32:18.053630 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:18.053531 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6da871e-dedf-412b-bbad-4257087b71a7-istio-token\") pod \"e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d\" (UID: \"f6da871e-dedf-412b-bbad-4257087b71a7\") " pod="openshift-ingress/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d" Nov 02 01:32:18.054239 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:18.054212 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6da871e-dedf-412b-bbad-4257087b71a7-istio-podinfo\") pod \"e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d\" (UID: \"f6da871e-dedf-412b-bbad-4257087b71a7\") " pod="openshift-ingress/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d" Nov 02 01:32:18.054982 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:18.054962 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6da871e-dedf-412b-bbad-4257087b71a7-workload-socket\") pod \"e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d\" (UID: \"f6da871e-dedf-412b-bbad-4257087b71a7\") " pod="openshift-ingress/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d" Nov 02 01:32:18.055688 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:18.055670 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6da871e-dedf-412b-bbad-4257087b71a7-istio-envoy\") pod \"e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d\" (UID: \"f6da871e-dedf-412b-bbad-4257087b71a7\") " pod="openshift-ingress/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d" Nov 02 01:32:18.056157 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:18.056139 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6da871e-dedf-412b-bbad-4257087b71a7-istiod-ca-cert\") pod \"e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d\" (UID: \"f6da871e-dedf-412b-bbad-4257087b71a7\") " pod="openshift-ingress/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d" Nov 02 01:32:18.056994 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:18.056976 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6da871e-dedf-412b-bbad-4257087b71a7-workload-certs\") pod \"e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d\" (UID: \"f6da871e-dedf-412b-bbad-4257087b71a7\") " pod="openshift-ingress/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d" Nov 02 01:32:18.062509 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:18.062478 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvf22\" (UniqueName: \"kubernetes.io/projected/f6da871e-dedf-412b-bbad-4257087b71a7-kube-api-access-dvf22\") pod \"e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d\" (UID: \"f6da871e-dedf-412b-bbad-4257087b71a7\") " pod="openshift-ingress/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d" Nov 02 01:32:18.062749 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:18.062729 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6da871e-dedf-412b-bbad-4257087b71a7-istio-data\") pod \"e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d\" (UID: \"f6da871e-dedf-412b-bbad-4257087b71a7\") " pod="openshift-ingress/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d" Nov 02 01:32:18.063188 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:18.063168 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6da871e-dedf-412b-bbad-4257087b71a7-credential-socket\") pod \"e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d\" (UID: \"f6da871e-dedf-412b-bbad-4257087b71a7\") " pod="openshift-ingress/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d" Nov 02 01:32:18.064289 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:18.064265 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6da871e-dedf-412b-bbad-4257087b71a7-credential-socket\") pod \"e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d\" (UID: \"f6da871e-dedf-412b-bbad-4257087b71a7\") " pod="openshift-ingress/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d" Nov 02 01:32:18.064456 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:18.056900 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6da871e-dedf-412b-bbad-4257087b71a7-istiod-ca-cert\") pod \"e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d\" (UID: \"f6da871e-dedf-412b-bbad-4257087b71a7\") " pod="openshift-ingress/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d" Nov 02 01:32:18.064832 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:18.055451 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6da871e-dedf-412b-bbad-4257087b71a7-workload-socket\") pod \"e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d\" (UID: \"f6da871e-dedf-412b-bbad-4257087b71a7\") " pod="openshift-ingress/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d" Nov 02 01:32:18.071048 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:18.070978 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6da871e-dedf-412b-bbad-4257087b71a7-istio-data\") pod \"e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d\" (UID: \"f6da871e-dedf-412b-bbad-4257087b71a7\") " pod="openshift-ingress/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d" Nov 02 01:32:18.071215 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:18.057330 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6da871e-dedf-412b-bbad-4257087b71a7-workload-certs\") pod \"e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d\" (UID: \"f6da871e-dedf-412b-bbad-4257087b71a7\") " pod="openshift-ingress/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d" Nov 02 01:32:18.074244 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:18.074198 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6da871e-dedf-412b-bbad-4257087b71a7-istio-token\") pod \"e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d\" (UID: \"f6da871e-dedf-412b-bbad-4257087b71a7\") " pod="openshift-ingress/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d" Nov 02 01:32:18.077538 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:18.077496 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvf22\" (UniqueName: \"kubernetes.io/projected/f6da871e-dedf-412b-bbad-4257087b71a7-kube-api-access-dvf22\") pod \"e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d\" (UID: \"f6da871e-dedf-412b-bbad-4257087b71a7\") " pod="openshift-ingress/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d" Nov 02 01:32:18.079175 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:18.079153 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6da871e-dedf-412b-bbad-4257087b71a7-istio-podinfo\") pod \"e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d\" (UID: \"f6da871e-dedf-412b-bbad-4257087b71a7\") " pod="openshift-ingress/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d" Nov 02 01:32:18.079817 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:18.079775 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6da871e-dedf-412b-bbad-4257087b71a7-istio-envoy\") pod \"e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d\" (UID: \"f6da871e-dedf-412b-bbad-4257087b71a7\") " pod="openshift-ingress/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d" Nov 02 01:32:18.155400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:18.154519 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d" Nov 02 01:32:18.522062 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:18.522020 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d"] Nov 02 01:32:18.522270 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:32:18.522244 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6da871e_dedf_412b_bbad_4257087b71a7.slice/crio-e1c88751a02899b61415f5f398386bc27b82d486b4cce4e3902af6e34ef86320 WatchSource:0}: Error finding container e1c88751a02899b61415f5f398386bc27b82d486b4cce4e3902af6e34ef86320: Status 404 returned error can't find the container with id e1c88751a02899b61415f5f398386bc27b82d486b4cce4e3902af6e34ef86320 Nov 02 01:32:18.525359 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:18.525317 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:32:18.525482 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:18.525460 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:32:18.525527 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:18.525515 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:32:19.554877 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:19.551930 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d" event={"ID":"f6da871e-dedf-412b-bbad-4257087b71a7","Type":"ContainerStarted","Data":"c6f8e2e24dbd17f2d6bae654cf655359df1ac226eef7c6366564fb8f6182f828"} Nov 02 01:32:19.554877 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:19.552005 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d" event={"ID":"f6da871e-dedf-412b-bbad-4257087b71a7","Type":"ContainerStarted","Data":"e1c88751a02899b61415f5f398386bc27b82d486b4cce4e3902af6e34ef86320"} Nov 02 01:32:19.565774 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:19.565661 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d" podStartSLOduration=2.56563947 podStartE2EDuration="2.56563947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:32:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:32:19.561611902 +0000 UTC m=+1044.182617130" watchObservedRunningTime="2026-11-02 01:32:19.56563947 +0000 UTC m=+1044.186644683" Nov 02 01:32:20.155320 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:20.155285 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d" Nov 02 01:32:20.162138 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:20.162100 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d" Nov 02 01:32:20.537862 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:20.537773 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d" Nov 02 01:32:20.538716 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:20.538696 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d" Nov 02 01:32:23.131904 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:23.131869 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-c22a0d-569bff79c5-8qj76"] Nov 02 01:32:23.132630 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:23.132610 4427 cpu_manager.go:414] "containerMap: removing container" podUID="a221bfd5-295a-456b-ae11-afe35719f55d" containerName="istio-proxy" Nov 02 01:32:23.132630 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:23.132630 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a221bfd5-295a-456b-ae11-afe35719f55d" containerName="istio-proxy" Nov 02 01:32:23.132744 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:23.132714 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="a221bfd5-295a-456b-ae11-afe35719f55d" containerName="istio-proxy" Nov 02 01:32:23.136358 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:23.136341 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c22a0d-569bff79c5-8qj76" Nov 02 01:32:23.144296 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:23.144268 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-c22a0d-dockercfg-vtj5b\"" Nov 02 01:32:23.147433 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:23.144917 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-c22a0d\"" Nov 02 01:32:23.150968 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:23.150941 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-c22a0d-569bff79c5-8qj76"] Nov 02 01:32:23.172827 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:23.172776 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c22a0d-7bd45b488c-gr9gq"] Nov 02 01:32:23.174150 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:23.174120 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l69dk\" (UniqueName: \"kubernetes.io/projected/23707aeb-a006-4049-bca4-074bbd446ca7-kube-api-access-l69dk\") pod \"payload-processing-e2e-shared-b-w4-c22a0d-569bff79c5-8qj76\" (UID: \"23707aeb-a006-4049-bca4-074bbd446ca7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c22a0d-569bff79c5-8qj76" Nov 02 01:32:23.175863 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:23.174216 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23707aeb-a006-4049-bca4-074bbd446ca7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-c22a0d-569bff79c5-8qj76\" (UID: \"23707aeb-a006-4049-bca4-074bbd446ca7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c22a0d-569bff79c5-8qj76" Nov 02 01:32:23.179221 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:23.179199 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c22a0d-7bd45b488c-gr9gq" Nov 02 01:32:23.179344 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:23.179292 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c22a0d-7bd45b488c-gr9gq"] Nov 02 01:32:23.361414 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:23.352983 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l69dk\" (UniqueName: \"kubernetes.io/projected/23707aeb-a006-4049-bca4-074bbd446ca7-kube-api-access-l69dk\") pod \"payload-processing-e2e-shared-b-w4-c22a0d-569bff79c5-8qj76\" (UID: \"23707aeb-a006-4049-bca4-074bbd446ca7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c22a0d-569bff79c5-8qj76" Nov 02 01:32:23.361414 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:23.353098 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23707aeb-a006-4049-bca4-074bbd446ca7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-c22a0d-569bff79c5-8qj76\" (UID: \"23707aeb-a006-4049-bca4-074bbd446ca7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c22a0d-569bff79c5-8qj76" Nov 02 01:32:23.361414 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:23.353207 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8nbm\" (UniqueName: \"kubernetes.io/projected/0ed6780b-6074-4ce4-b9c0-44253b32955c-kube-api-access-j8nbm\") pod \"payload-pre-processing-e2e-shared-b-w4-c22a0d-7bd45b488c-gr9gq\" (UID: \"0ed6780b-6074-4ce4-b9c0-44253b32955c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c22a0d-7bd45b488c-gr9gq" Nov 02 01:32:23.361414 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:23.353313 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ed6780b-6074-4ce4-b9c0-44253b32955c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-c22a0d-7bd45b488c-gr9gq\" (UID: \"0ed6780b-6074-4ce4-b9c0-44253b32955c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c22a0d-7bd45b488c-gr9gq" Nov 02 01:32:23.386453 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:23.382227 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23707aeb-a006-4049-bca4-074bbd446ca7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-c22a0d-569bff79c5-8qj76\" (UID: \"23707aeb-a006-4049-bca4-074bbd446ca7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c22a0d-569bff79c5-8qj76" Nov 02 01:32:23.421273 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:23.421156 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l69dk\" (UniqueName: \"kubernetes.io/projected/23707aeb-a006-4049-bca4-074bbd446ca7-kube-api-access-l69dk\") pod \"payload-processing-e2e-shared-b-w4-c22a0d-569bff79c5-8qj76\" (UID: \"23707aeb-a006-4049-bca4-074bbd446ca7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c22a0d-569bff79c5-8qj76" Nov 02 01:32:23.457987 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:23.457684 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c22a0d-569bff79c5-8qj76" Nov 02 01:32:23.486683 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:23.483003 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ed6780b-6074-4ce4-b9c0-44253b32955c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-c22a0d-7bd45b488c-gr9gq\" (UID: \"0ed6780b-6074-4ce4-b9c0-44253b32955c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c22a0d-7bd45b488c-gr9gq" Nov 02 01:32:23.486683 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:23.484802 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ed6780b-6074-4ce4-b9c0-44253b32955c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-c22a0d-7bd45b488c-gr9gq\" (UID: \"0ed6780b-6074-4ce4-b9c0-44253b32955c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c22a0d-7bd45b488c-gr9gq" Nov 02 01:32:23.486683 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:23.484986 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8nbm\" (UniqueName: \"kubernetes.io/projected/0ed6780b-6074-4ce4-b9c0-44253b32955c-kube-api-access-j8nbm\") pod \"payload-pre-processing-e2e-shared-b-w4-c22a0d-7bd45b488c-gr9gq\" (UID: \"0ed6780b-6074-4ce4-b9c0-44253b32955c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c22a0d-7bd45b488c-gr9gq" Nov 02 01:32:23.516399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:23.516351 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8nbm\" (UniqueName: \"kubernetes.io/projected/0ed6780b-6074-4ce4-b9c0-44253b32955c-kube-api-access-j8nbm\") pod \"payload-pre-processing-e2e-shared-b-w4-c22a0d-7bd45b488c-gr9gq\" (UID: \"0ed6780b-6074-4ce4-b9c0-44253b32955c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c22a0d-7bd45b488c-gr9gq" Nov 02 01:32:23.802542 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:23.795117 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c22a0d-7bd45b488c-gr9gq" Nov 02 01:32:23.825835 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:23.825791 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-c22a0d-569bff79c5-8qj76"] Nov 02 01:32:24.094211 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:24.094185 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c22a0d-7bd45b488c-gr9gq"] Nov 02 01:32:24.094788 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:32:24.094768 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ed6780b_6074_4ce4_b9c0_44253b32955c.slice/crio-650e767fd9c9231ea0b9605a4c571309b638dcee92b7761f9283c52fd0cceaec WatchSource:0}: Error finding container 650e767fd9c9231ea0b9605a4c571309b638dcee92b7761f9283c52fd0cceaec: Status 404 returned error can't find the container with id 650e767fd9c9231ea0b9605a4c571309b638dcee92b7761f9283c52fd0cceaec Nov 02 01:32:24.583337 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:24.583297 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c22a0d-7bd45b488c-gr9gq" event={"ID":"0ed6780b-6074-4ce4-b9c0-44253b32955c","Type":"ContainerStarted","Data":"6f0c66960a98bd6033aa4054cfb10407e08744e16b2c93963d60e3bbc0d64da1"} Nov 02 01:32:24.583891 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:24.583369 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c22a0d-7bd45b488c-gr9gq" event={"ID":"0ed6780b-6074-4ce4-b9c0-44253b32955c","Type":"ContainerStarted","Data":"650e767fd9c9231ea0b9605a4c571309b638dcee92b7761f9283c52fd0cceaec"} Nov 02 01:32:24.583891 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:24.583550 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c22a0d-7bd45b488c-gr9gq" Nov 02 01:32:24.585276 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:24.585246 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c22a0d-569bff79c5-8qj76" event={"ID":"23707aeb-a006-4049-bca4-074bbd446ca7","Type":"ContainerStarted","Data":"d7476636108306de61328f8070f68e0a4ad18b97a1e992c1c59e3289e3841d81"} Nov 02 01:32:24.585523 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:24.585297 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c22a0d-569bff79c5-8qj76" event={"ID":"23707aeb-a006-4049-bca4-074bbd446ca7","Type":"ContainerStarted","Data":"cd6f54a3f22e95c234511cbf0069f122b2f970f17db3d8621328ddec9bd532f6"} Nov 02 01:32:24.585523 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:24.585357 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c22a0d-569bff79c5-8qj76" Nov 02 01:32:24.605948 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:24.605896 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c22a0d-7bd45b488c-gr9gq" podStartSLOduration=1.6058765689999999 podStartE2EDuration="1.605876569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:32:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:32:24.603065828 +0000 UTC m=+1049.224071035" watchObservedRunningTime="2026-11-02 01:32:24.605876569 +0000 UTC m=+1049.226881780" Nov 02 01:32:31.585043 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:32:31.584996 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:32:35.596194 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:35.596165 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c22a0d-569bff79c5-8qj76" Nov 02 01:32:35.596715 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:35.596678 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c22a0d-7bd45b488c-gr9gq" Nov 02 01:32:35.618629 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:35.618579 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c22a0d-569bff79c5-8qj76" podStartSLOduration=12.61856446 podStartE2EDuration="12.61856446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:32:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:32:24.627335581 +0000 UTC m=+1049.248340796" watchObservedRunningTime="2026-11-02 01:32:35.61856446 +0000 UTC m=+1060.239569675" Nov 02 01:32:41.961169 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:41.961133 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-50537f4f-67d6cc6b87-kqzbs"] Nov 02 01:32:41.965627 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:41.965604 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-50537f4f-67d6cc6b87-kqzbs" Nov 02 01:32:41.972564 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:41.972538 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-50537f4f\"" Nov 02 01:32:41.976738 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:41.976632 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-50537f4f-67d6cc6b87-kqzbs"] Nov 02 01:32:42.103424 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:42.103356 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwkfl\" (UniqueName: \"kubernetes.io/projected/d7877d44-4cab-4c19-a7f1-d761b2be0bb6-kube-api-access-vwkfl\") pod \"payload-processing-e2e-mt-50537f4f-67d6cc6b87-kqzbs\" (UID: \"d7877d44-4cab-4c19-a7f1-d761b2be0bb6\") " pod="openshift-ingress/payload-processing-e2e-mt-50537f4f-67d6cc6b87-kqzbs" Nov 02 01:32:42.103641 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:42.103565 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7877d44-4cab-4c19-a7f1-d761b2be0bb6-plugins-config-volume\") pod \"payload-processing-e2e-mt-50537f4f-67d6cc6b87-kqzbs\" (UID: \"d7877d44-4cab-4c19-a7f1-d761b2be0bb6\") " pod="openshift-ingress/payload-processing-e2e-mt-50537f4f-67d6cc6b87-kqzbs" Nov 02 01:32:42.216487 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:42.212087 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwkfl\" (UniqueName: \"kubernetes.io/projected/d7877d44-4cab-4c19-a7f1-d761b2be0bb6-kube-api-access-vwkfl\") pod \"payload-processing-e2e-mt-50537f4f-67d6cc6b87-kqzbs\" (UID: \"d7877d44-4cab-4c19-a7f1-d761b2be0bb6\") " pod="openshift-ingress/payload-processing-e2e-mt-50537f4f-67d6cc6b87-kqzbs" Nov 02 01:32:42.216487 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:42.212232 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7877d44-4cab-4c19-a7f1-d761b2be0bb6-plugins-config-volume\") pod \"payload-processing-e2e-mt-50537f4f-67d6cc6b87-kqzbs\" (UID: \"d7877d44-4cab-4c19-a7f1-d761b2be0bb6\") " pod="openshift-ingress/payload-processing-e2e-mt-50537f4f-67d6cc6b87-kqzbs" Nov 02 01:32:42.240423 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:42.229721 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7877d44-4cab-4c19-a7f1-d761b2be0bb6-plugins-config-volume\") pod \"payload-processing-e2e-mt-50537f4f-67d6cc6b87-kqzbs\" (UID: \"d7877d44-4cab-4c19-a7f1-d761b2be0bb6\") " pod="openshift-ingress/payload-processing-e2e-mt-50537f4f-67d6cc6b87-kqzbs" Nov 02 01:32:42.260488 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:42.249103 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwkfl\" (UniqueName: \"kubernetes.io/projected/d7877d44-4cab-4c19-a7f1-d761b2be0bb6-kube-api-access-vwkfl\") pod \"payload-processing-e2e-mt-50537f4f-67d6cc6b87-kqzbs\" (UID: \"d7877d44-4cab-4c19-a7f1-d761b2be0bb6\") " pod="openshift-ingress/payload-processing-e2e-mt-50537f4f-67d6cc6b87-kqzbs" Nov 02 01:32:42.291455 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:42.288691 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-50537f4f-67d6cc6b87-kqzbs" Nov 02 01:32:42.637069 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:42.636954 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-50537f4f-67d6cc6b87-kqzbs"] Nov 02 01:32:42.638675 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:32:42.638646 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7877d44_4cab_4c19_a7f1_d761b2be0bb6.slice/crio-142845e727f9d33122761b783c5bf0ec972db8b935283ccdeb3d39d975704cd6 WatchSource:0}: Error finding container 142845e727f9d33122761b783c5bf0ec972db8b935283ccdeb3d39d975704cd6: Status 404 returned error can't find the container with id 142845e727f9d33122761b783c5bf0ec972db8b935283ccdeb3d39d975704cd6 Nov 02 01:32:42.785267 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:42.785230 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-50537f4f-67d6cc6b87-kqzbs" event={"ID":"d7877d44-4cab-4c19-a7f1-d761b2be0bb6","Type":"ContainerStarted","Data":"e0c0a7f318355c0d5c099cfdea9e8e9c2f2e237a8ac65493a667e86a2d6287c5"} Nov 02 01:32:42.785477 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:42.785279 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-50537f4f-67d6cc6b87-kqzbs" event={"ID":"d7877d44-4cab-4c19-a7f1-d761b2be0bb6","Type":"ContainerStarted","Data":"142845e727f9d33122761b783c5bf0ec972db8b935283ccdeb3d39d975704cd6"} Nov 02 01:32:42.785477 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:42.785312 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-50537f4f-67d6cc6b87-kqzbs" Nov 02 01:32:42.805229 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:42.805173 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-50537f4f-67d6cc6b87-kqzbs" podStartSLOduration=1.8051588170000001 podStartE2EDuration="1.805158817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:32:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:32:42.803717103 +0000 UTC m=+1067.424722331" watchObservedRunningTime="2026-11-02 01:32:42.805158817 +0000 UTC m=+1067.426164070" Nov 02 01:32:47.285537 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:47.285498 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-616e3ed7-bc54d9469-hzxbz"] Nov 02 01:32:52.285880 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:52.285814 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-616e3ed7-bc54d9469-hzxbz" podUID="64ff5b5c-cc7a-4a09-81c8-17987708c2ee" containerName="payload-processing" containerID="cri-o://41709546bc5622c3baa7771c8ef6aa12198e264e1b263931c84f96075e58951b" gracePeriod=25 Nov 02 01:32:52.610451 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:52.609413 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-616e3ed7-bc54d9469-hzxbz" Nov 02 01:32:52.779339 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:52.779307 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64ff5b5c-cc7a-4a09-81c8-17987708c2ee-kube-api-access-h2sh6\" (UniqueName: \"kubernetes.io/projected/64ff5b5c-cc7a-4a09-81c8-17987708c2ee-kube-api-access-h2sh6\") pod \"64ff5b5c-cc7a-4a09-81c8-17987708c2ee\" (UID: \"64ff5b5c-cc7a-4a09-81c8-17987708c2ee\") " Nov 02 01:32:52.779584 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:52.779527 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64ff5b5c-cc7a-4a09-81c8-17987708c2ee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ff5b5c-cc7a-4a09-81c8-17987708c2ee-plugins-config-volume\") pod \"64ff5b5c-cc7a-4a09-81c8-17987708c2ee\" (UID: \"64ff5b5c-cc7a-4a09-81c8-17987708c2ee\") " Nov 02 01:32:52.779978 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:52.779948 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64ff5b5c-cc7a-4a09-81c8-17987708c2ee-plugins-config-volume" pod "64ff5b5c-cc7a-4a09-81c8-17987708c2ee" (UID: "64ff5b5c-cc7a-4a09-81c8-17987708c2ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:32:52.780464 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:52.780438 4427 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ff5b5c-cc7a-4a09-81c8-17987708c2ee-plugins-config-volume\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:32:52.782062 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:52.782031 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64ff5b5c-cc7a-4a09-81c8-17987708c2ee-kube-api-access-h2sh6" pod "64ff5b5c-cc7a-4a09-81c8-17987708c2ee" (UID: "64ff5b5c-cc7a-4a09-81c8-17987708c2ee"). InnerVolumeSpecName "kube-api-access-h2sh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:32:52.887626 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:52.887576 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2sh6\" (UniqueName: \"kubernetes.io/projected/64ff5b5c-cc7a-4a09-81c8-17987708c2ee-kube-api-access-h2sh6\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:32:52.896223 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:52.896192 4427 generic.go:356] "Generic (PLEG): container finished" podID="64ff5b5c-cc7a-4a09-81c8-17987708c2ee" containerID="41709546bc5622c3baa7771c8ef6aa12198e264e1b263931c84f96075e58951b" exitCode=0 Nov 02 01:32:52.896437 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:52.896251 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-616e3ed7-bc54d9469-hzxbz" Nov 02 01:32:52.896437 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:52.896271 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-616e3ed7-bc54d9469-hzxbz" event={"ID":"64ff5b5c-cc7a-4a09-81c8-17987708c2ee","Type":"ContainerDied","Data":"41709546bc5622c3baa7771c8ef6aa12198e264e1b263931c84f96075e58951b"} Nov 02 01:32:52.896437 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:52.896329 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-616e3ed7-bc54d9469-hzxbz" event={"ID":"64ff5b5c-cc7a-4a09-81c8-17987708c2ee","Type":"ContainerDied","Data":"3a679fbc3a63988c0dd9e07e92ad8272a6892cf914b051e78f9b6fcc50125f0e"} Nov 02 01:32:52.896437 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:52.896351 4427 scope.go:122] "RemoveContainer" containerID="41709546bc5622c3baa7771c8ef6aa12198e264e1b263931c84f96075e58951b" Nov 02 01:32:52.908158 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:52.908138 4427 scope.go:122] "RemoveContainer" containerID="41709546bc5622c3baa7771c8ef6aa12198e264e1b263931c84f96075e58951b" Nov 02 01:32:52.908528 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:32:52.908506 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41709546bc5622c3baa7771c8ef6aa12198e264e1b263931c84f96075e58951b\": container with ID starting with 41709546bc5622c3baa7771c8ef6aa12198e264e1b263931c84f96075e58951b not found: ID does not exist" containerID="41709546bc5622c3baa7771c8ef6aa12198e264e1b263931c84f96075e58951b" Nov 02 01:32:52.908644 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:52.908537 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41709546bc5622c3baa7771c8ef6aa12198e264e1b263931c84f96075e58951b"} err="failed to get container status \"41709546bc5622c3baa7771c8ef6aa12198e264e1b263931c84f96075e58951b\": rpc error: code = NotFound desc = could not find container \"41709546bc5622c3baa7771c8ef6aa12198e264e1b263931c84f96075e58951b\": container with ID starting with 41709546bc5622c3baa7771c8ef6aa12198e264e1b263931c84f96075e58951b not found: ID does not exist" Nov 02 01:32:52.921897 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:52.921866 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-616e3ed7-bc54d9469-hzxbz"] Nov 02 01:32:52.926122 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:52.926083 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-616e3ed7-bc54d9469-hzxbz"] Nov 02 01:32:53.535897 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:53.535866 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64ff5b5c-cc7a-4a09-81c8-17987708c2ee" path="/var/lib/kubelet/pods/64ff5b5c-cc7a-4a09-81c8-17987708c2ee/volumes" Nov 02 01:32:53.796980 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:32:53.796889 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-50537f4f-67d6cc6b87-kqzbs" Nov 02 01:33:01.595581 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:33:01.595542 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:33:12.391402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:12.389461 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2ebe99-c55c4c955-fxcm6"] Nov 02 01:33:12.391402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:12.390742 4427 cpu_manager.go:414] "containerMap: removing container" podUID="64ff5b5c-cc7a-4a09-81c8-17987708c2ee" containerName="payload-processing" Nov 02 01:33:12.391402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:12.390767 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64ff5b5c-cc7a-4a09-81c8-17987708c2ee" containerName="payload-processing" Nov 02 01:33:12.391402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:12.390950 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="64ff5b5c-cc7a-4a09-81c8-17987708c2ee" containerName="payload-processing" Nov 02 01:33:12.627530 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:12.624718 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2ebe99-c55c4c955-fxcm6" Nov 02 01:33:12.645478 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:12.644348 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-2ebe99\"" Nov 02 01:33:12.645478 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:12.644551 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-2ebe99-dockercfg-mfzld\"" Nov 02 01:33:12.689075 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:12.689041 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2ebe99-c55c4c955-fxcm6"] Nov 02 01:33:12.794288 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:12.794230 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnzv2\" (UniqueName: \"kubernetes.io/projected/79a5f4f6-4fad-4cfa-b5e7-562db4195d4a-kube-api-access-fnzv2\") pod \"payload-processing-e2e-api-a-2ebe99-c55c4c955-fxcm6\" (UID: \"79a5f4f6-4fad-4cfa-b5e7-562db4195d4a\") " pod="openshift-ingress/payload-processing-e2e-api-a-2ebe99-c55c4c955-fxcm6" Nov 02 01:33:12.794689 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:12.794643 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79a5f4f6-4fad-4cfa-b5e7-562db4195d4a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2ebe99-c55c4c955-fxcm6\" (UID: \"79a5f4f6-4fad-4cfa-b5e7-562db4195d4a\") " pod="openshift-ingress/payload-processing-e2e-api-a-2ebe99-c55c4c955-fxcm6" Nov 02 01:33:12.900001 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:12.899908 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnzv2\" (UniqueName: \"kubernetes.io/projected/79a5f4f6-4fad-4cfa-b5e7-562db4195d4a-kube-api-access-fnzv2\") pod \"payload-processing-e2e-api-a-2ebe99-c55c4c955-fxcm6\" (UID: \"79a5f4f6-4fad-4cfa-b5e7-562db4195d4a\") " pod="openshift-ingress/payload-processing-e2e-api-a-2ebe99-c55c4c955-fxcm6" Nov 02 01:33:12.900328 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:12.900302 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79a5f4f6-4fad-4cfa-b5e7-562db4195d4a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2ebe99-c55c4c955-fxcm6\" (UID: \"79a5f4f6-4fad-4cfa-b5e7-562db4195d4a\") " pod="openshift-ingress/payload-processing-e2e-api-a-2ebe99-c55c4c955-fxcm6" Nov 02 01:33:12.928403 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:12.901428 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79a5f4f6-4fad-4cfa-b5e7-562db4195d4a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2ebe99-c55c4c955-fxcm6\" (UID: \"79a5f4f6-4fad-4cfa-b5e7-562db4195d4a\") " pod="openshift-ingress/payload-processing-e2e-api-a-2ebe99-c55c4c955-fxcm6" Nov 02 01:33:12.958458 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:12.949554 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnzv2\" (UniqueName: \"kubernetes.io/projected/79a5f4f6-4fad-4cfa-b5e7-562db4195d4a-kube-api-access-fnzv2\") pod \"payload-processing-e2e-api-a-2ebe99-c55c4c955-fxcm6\" (UID: \"79a5f4f6-4fad-4cfa-b5e7-562db4195d4a\") " pod="openshift-ingress/payload-processing-e2e-api-a-2ebe99-c55c4c955-fxcm6" Nov 02 01:33:13.057566 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:13.057527 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" podUID="eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 01:33:13.076550 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:13.069829 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" podUID="cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:33:13.085406 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:13.070674 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" podUID="47a9069f-1ffa-4c83-be8b-5e62f1d3275a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:33:13.098075 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:13.085131 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2ebe99-c55c4c955-fxcm6" Nov 02 01:33:13.364399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:13.343944 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" podUID="ea61971c-5a72-4609-b5a0-58a74b324bb7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:33:13.634403 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:13.633806 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" podUID="c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:33:13.929493 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:13.929360 4427 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" podUID="706df1bd-19cf-4ab9-9aaa-3c12c97ccf45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:33:13.935489 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:13.932419 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" podUID="fb533b5f-3525-454a-8ce2-ea9246194f28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Nov 02 01:33:13.936544 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:13.936513 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" podUID="366c78ee-4aaf-4b7c-84b4-17028538cb1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:33:14.076321 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:14.076280 4427 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" podUID="47a9069f-1ffa-4c83-be8b-5e62f1d3275a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": context deadline exceeded" Nov 02 01:33:14.085010 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:14.084970 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" podUID="cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:33:14.108439 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:14.099723 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" Nov 02 01:33:14.108439 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:14.101095 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" podUID="47a9069f-1ffa-4c83-be8b-5e62f1d3275a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 02 01:33:14.170728 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:14.129127 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" Nov 02 01:33:14.362216 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:14.355663 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" podUID="f67c954a-ce5b-41d9-8d99-7d126bd89d59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:33:14.368324 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:14.367266 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" podUID="7085c0ab-7ee3-4665-abad-2977c25f68ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:33:14.438180 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:14.438140 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" podUID="353a8a41-21ee-4470-861c-b53833beb106" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:33:14.469869 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:14.467563 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" podUID="706df1bd-19cf-4ab9-9aaa-3c12c97ccf45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 01:33:14.559298 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:14.559269 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" Nov 02 01:33:14.712458 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:14.703982 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" podUID="d73ae484-a9b8-4565-a346-0d803a1981ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:33:14.712458 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:14.704349 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" podUID="c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:33:14.740188 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:14.721170 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" Nov 02 01:33:14.837658 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:14.837628 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" Nov 02 01:33:14.859368 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:14.859329 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" podUID="9c056798-2cf8-47b1-9829-67cebda7a248" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 01:33:14.934916 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:14.934727 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" podUID="fb533b5f-3525-454a-8ce2-ea9246194f28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Nov 02 01:33:14.935607 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:14.935518 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" Nov 02 01:33:14.956673 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:14.936797 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" podUID="366c78ee-4aaf-4b7c-84b4-17028538cb1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:33:14.956673 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:14.937081 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" Nov 02 01:33:15.082314 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:15.071888 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" Nov 02 01:33:15.175058 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:15.172170 4427 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" podUID="7085c0ab-7ee3-4665-abad-2977c25f68ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:33:15.201541 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:15.201497 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2ebe99-c55c4c955-fxcm6"] Nov 02 01:33:15.238397 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:15.236256 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" Nov 02 01:33:15.280533 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:15.270977 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" Nov 02 01:33:15.944524 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:15.944485 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2ebe99-c55c4c955-fxcm6" event={"ID":"79a5f4f6-4fad-4cfa-b5e7-562db4195d4a","Type":"ContainerStarted","Data":"d7bc878094983d7327eca9233efcb948fdabbab2c77f2a05b3d42222665d5a81"} Nov 02 01:33:15.945141 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:15.945113 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2ebe99-c55c4c955-fxcm6" event={"ID":"79a5f4f6-4fad-4cfa-b5e7-562db4195d4a","Type":"ContainerStarted","Data":"aa830c846fc612934032dded73c4c73daadbdd53a031f906eb9e27584479cd0e"} Nov 02 01:33:16.949185 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:16.949150 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-2ebe99-c55c4c955-fxcm6" Nov 02 01:33:21.475246 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:21.474702 4427 patch_prober.go:29] interesting pod/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": read tcp 10.133.0.2:37020->10.133.0.94:15021: read: connection reset by peer" start-of-body= Nov 02 01:33:21.475246 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:21.474784 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d" podUID="f6da871e-dedf-412b-bbad-4257087b71a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": read tcp 10.133.0.2:37020->10.133.0.94:15021: read: connection reset by peer" Nov 02 01:33:21.579609 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:21.579560 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-2ebe99-c55c4c955-fxcm6" podStartSLOduration=9.579545269 podStartE2EDuration="9.579545269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:33:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:33:15.993721355 +0000 UTC m=+1100.614726577" watchObservedRunningTime="2026-11-02 01:33:21.579545269 +0000 UTC m=+1106.200550538" Nov 02 01:33:21.579792 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:21.579708 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4"] Nov 02 01:33:21.584253 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:21.584233 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4" Nov 02 01:33:21.590761 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:21.590629 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-50537f4f-openshift-default-dockercfg-2fm44\"" Nov 02 01:33:21.599007 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:21.598283 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4"] Nov 02 01:33:21.616328 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:21.616063 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8eafff6-528c-4e61-adff-8aed03dea982-istio-data\") pod \"e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4\" (UID: \"f8eafff6-528c-4e61-adff-8aed03dea982\") " pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4" Nov 02 01:33:21.616328 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:21.616106 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8eafff6-528c-4e61-adff-8aed03dea982-istio-podinfo\") pod \"e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4\" (UID: \"f8eafff6-528c-4e61-adff-8aed03dea982\") " pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4" Nov 02 01:33:21.616328 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:21.616213 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8eafff6-528c-4e61-adff-8aed03dea982-workload-certs\") pod \"e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4\" (UID: \"f8eafff6-528c-4e61-adff-8aed03dea982\") " pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4" Nov 02 01:33:21.616328 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:21.616268 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m86xd\" (UniqueName: \"kubernetes.io/projected/f8eafff6-528c-4e61-adff-8aed03dea982-kube-api-access-m86xd\") pod \"e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4\" (UID: \"f8eafff6-528c-4e61-adff-8aed03dea982\") " pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4" Nov 02 01:33:21.616328 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:21.616331 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8eafff6-528c-4e61-adff-8aed03dea982-credential-socket\") pod \"e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4\" (UID: \"f8eafff6-528c-4e61-adff-8aed03dea982\") " pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4" Nov 02 01:33:21.616793 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:21.616458 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8eafff6-528c-4e61-adff-8aed03dea982-istio-token\") pod \"e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4\" (UID: \"f8eafff6-528c-4e61-adff-8aed03dea982\") " pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4" Nov 02 01:33:21.616793 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:21.616495 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8eafff6-528c-4e61-adff-8aed03dea982-workload-socket\") pod \"e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4\" (UID: \"f8eafff6-528c-4e61-adff-8aed03dea982\") " pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4" Nov 02 01:33:21.616793 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:21.616568 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8eafff6-528c-4e61-adff-8aed03dea982-istio-envoy\") pod \"e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4\" (UID: \"f8eafff6-528c-4e61-adff-8aed03dea982\") " pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4" Nov 02 01:33:21.616793 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:21.616740 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8eafff6-528c-4e61-adff-8aed03dea982-istiod-ca-cert\") pod \"e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4\" (UID: \"f8eafff6-528c-4e61-adff-8aed03dea982\") " pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4" Nov 02 01:33:21.725289 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:21.723694 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8eafff6-528c-4e61-adff-8aed03dea982-istiod-ca-cert\") pod \"e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4\" (UID: \"f8eafff6-528c-4e61-adff-8aed03dea982\") " pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4" Nov 02 01:33:21.725289 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:21.723819 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8eafff6-528c-4e61-adff-8aed03dea982-istio-data\") pod \"e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4\" (UID: \"f8eafff6-528c-4e61-adff-8aed03dea982\") " pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4" Nov 02 01:33:21.725289 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:21.723931 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8eafff6-528c-4e61-adff-8aed03dea982-istio-podinfo\") pod \"e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4\" (UID: \"f8eafff6-528c-4e61-adff-8aed03dea982\") " pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4" Nov 02 01:33:21.725289 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:21.724091 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8eafff6-528c-4e61-adff-8aed03dea982-workload-certs\") pod \"e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4\" (UID: \"f8eafff6-528c-4e61-adff-8aed03dea982\") " pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4" Nov 02 01:33:21.725289 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:21.724159 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m86xd\" (UniqueName: \"kubernetes.io/projected/f8eafff6-528c-4e61-adff-8aed03dea982-kube-api-access-m86xd\") pod \"e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4\" (UID: \"f8eafff6-528c-4e61-adff-8aed03dea982\") " pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4" Nov 02 01:33:21.725289 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:21.724218 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8eafff6-528c-4e61-adff-8aed03dea982-credential-socket\") pod \"e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4\" (UID: \"f8eafff6-528c-4e61-adff-8aed03dea982\") " pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4" Nov 02 01:33:21.725289 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:21.724280 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8eafff6-528c-4e61-adff-8aed03dea982-istio-token\") pod \"e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4\" (UID: \"f8eafff6-528c-4e61-adff-8aed03dea982\") " pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4" Nov 02 01:33:21.725289 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:21.724308 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8eafff6-528c-4e61-adff-8aed03dea982-workload-socket\") pod \"e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4\" (UID: \"f8eafff6-528c-4e61-adff-8aed03dea982\") " pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4" Nov 02 01:33:21.725289 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:21.724396 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8eafff6-528c-4e61-adff-8aed03dea982-istio-envoy\") pod \"e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4\" (UID: \"f8eafff6-528c-4e61-adff-8aed03dea982\") " pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4" Nov 02 01:33:21.725765 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:21.725295 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8eafff6-528c-4e61-adff-8aed03dea982-istiod-ca-cert\") pod \"e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4\" (UID: \"f8eafff6-528c-4e61-adff-8aed03dea982\") " pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4" Nov 02 01:33:21.725765 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:21.725651 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8eafff6-528c-4e61-adff-8aed03dea982-istio-data\") pod \"e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4\" (UID: \"f8eafff6-528c-4e61-adff-8aed03dea982\") " pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4" Nov 02 01:33:21.726294 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:21.726267 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8eafff6-528c-4e61-adff-8aed03dea982-credential-socket\") pod \"e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4\" (UID: \"f8eafff6-528c-4e61-adff-8aed03dea982\") " pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4" Nov 02 01:33:21.729001 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:21.726940 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8eafff6-528c-4e61-adff-8aed03dea982-workload-certs\") pod \"e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4\" (UID: \"f8eafff6-528c-4e61-adff-8aed03dea982\") " pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4" Nov 02 01:33:21.729493 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:21.729465 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8eafff6-528c-4e61-adff-8aed03dea982-istio-podinfo\") pod \"e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4\" (UID: \"f8eafff6-528c-4e61-adff-8aed03dea982\") " pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4" Nov 02 01:33:21.731589 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:21.731557 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8eafff6-528c-4e61-adff-8aed03dea982-workload-socket\") pod \"e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4\" (UID: \"f8eafff6-528c-4e61-adff-8aed03dea982\") " pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4" Nov 02 01:33:21.748550 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:21.744929 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8eafff6-528c-4e61-adff-8aed03dea982-istio-envoy\") pod \"e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4\" (UID: \"f8eafff6-528c-4e61-adff-8aed03dea982\") " pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4" Nov 02 01:33:21.748550 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:21.747803 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m86xd\" (UniqueName: \"kubernetes.io/projected/f8eafff6-528c-4e61-adff-8aed03dea982-kube-api-access-m86xd\") pod \"e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4\" (UID: \"f8eafff6-528c-4e61-adff-8aed03dea982\") " pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4" Nov 02 01:33:21.752906 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:21.748858 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8eafff6-528c-4e61-adff-8aed03dea982-istio-token\") pod \"e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4\" (UID: \"f8eafff6-528c-4e61-adff-8aed03dea982\") " pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4" Nov 02 01:33:21.908441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:21.907744 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4" Nov 02 01:33:22.016766 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:22.011454 4427 generic.go:356] "Generic (PLEG): container finished" podID="f6da871e-dedf-412b-bbad-4257087b71a7" containerID="c6f8e2e24dbd17f2d6bae654cf655359df1ac226eef7c6366564fb8f6182f828" exitCode=0 Nov 02 01:33:22.016766 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:22.011523 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d" event={"ID":"f6da871e-dedf-412b-bbad-4257087b71a7","Type":"ContainerDied","Data":"c6f8e2e24dbd17f2d6bae654cf655359df1ac226eef7c6366564fb8f6182f828"} Nov 02 01:33:22.016766 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:22.012014 4427 scope.go:122] "RemoveContainer" containerID="c6f8e2e24dbd17f2d6bae654cf655359df1ac226eef7c6366564fb8f6182f828" Nov 02 01:33:22.016766 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:22.014200 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:33:22.016766 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:22.014271 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:33:22.016766 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:22.014301 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:33:22.344249 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:22.344218 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4"] Nov 02 01:33:22.344785 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:22.344751 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:33:22.344887 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:22.344813 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:33:22.344887 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:22.344842 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:33:23.025325 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:23.025291 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d" event={"ID":"f6da871e-dedf-412b-bbad-4257087b71a7","Type":"ContainerStarted","Data":"0c04a2ada11dc080b9a1951ef18087dfc06e5e0b3e3b749d6fcac3e47c183669"} Nov 02 01:33:23.027388 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:23.027343 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4" event={"ID":"f8eafff6-528c-4e61-adff-8aed03dea982","Type":"ContainerStarted","Data":"a4acdb033cd6d67a4f1297a7d368d8477e0472d284b6ecc3984a7adf9d07717d"} Nov 02 01:33:23.027533 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:23.027414 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4" event={"ID":"f8eafff6-528c-4e61-adff-8aed03dea982","Type":"ContainerStarted","Data":"1c571d9175c498b37f7201d47eee07717c62e2777eb60e7127929e43c93d8b6c"} Nov 02 01:33:23.156027 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:23.155998 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d" Nov 02 01:33:23.156172 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:23.156117 4427 patch_prober.go:29] interesting pod/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 02 01:33:23.156235 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:23.156160 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d" podUID="f6da871e-dedf-412b-bbad-4257087b71a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 02 01:33:23.910919 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:23.910507 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4" Nov 02 01:33:23.911931 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:23.911891 4427 patch_prober.go:29] interesting pod/e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 02 01:33:23.912056 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:23.911958 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4" podUID="f8eafff6-528c-4e61-adff-8aed03dea982" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 02 01:33:24.184469 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:24.182834 4427 patch_prober.go:29] interesting pod/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 02 01:33:24.184469 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:24.182891 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d" podUID="f6da871e-dedf-412b-bbad-4257087b71a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 02 01:33:24.910013 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:24.909975 4427 patch_prober.go:29] interesting pod/e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 02 01:33:24.910313 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:24.910269 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4" podUID="f8eafff6-528c-4e61-adff-8aed03dea982" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 02 01:33:25.175680 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:25.175523 4427 patch_prober.go:29] interesting pod/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 02 01:33:25.175972 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:25.175941 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d" podUID="f6da871e-dedf-412b-bbad-4257087b71a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 02 01:33:25.916723 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:25.915879 4427 patch_prober.go:29] interesting pod/e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 02 01:33:25.916723 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:25.915941 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4" podUID="f8eafff6-528c-4e61-adff-8aed03dea982" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 02 01:33:26.156825 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:26.156783 4427 patch_prober.go:29] interesting pod/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 02 01:33:26.157037 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:26.156910 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d" podUID="f6da871e-dedf-412b-bbad-4257087b71a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 02 01:33:26.909404 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:26.908884 4427 patch_prober.go:29] interesting pod/e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 02 01:33:26.909404 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:26.908961 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4" podUID="f8eafff6-528c-4e61-adff-8aed03dea982" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 02 01:33:27.155406 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:27.155357 4427 patch_prober.go:29] interesting pod/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 02 01:33:27.155883 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:27.155436 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d" podUID="f6da871e-dedf-412b-bbad-4257087b71a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 02 01:33:27.913409 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:27.910162 4427 patch_prober.go:29] interesting pod/e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 02 01:33:27.913409 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:27.910237 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4" podUID="f8eafff6-528c-4e61-adff-8aed03dea982" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 02 01:33:28.002398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:28.001953 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-2ebe99-c55c4c955-fxcm6" Nov 02 01:33:28.029549 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:28.029495 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4" podStartSLOduration=7.029479891 podStartE2EDuration="7.029479891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:33:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:33:23.136923032 +0000 UTC m=+1107.757928269" watchObservedRunningTime="2026-11-02 01:33:28.029479891 +0000 UTC m=+1112.650485142" Nov 02 01:33:28.222470 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:28.197836 4427 patch_prober.go:29] interesting pod/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 02 01:33:28.222470 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:28.197899 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d" podUID="f6da871e-dedf-412b-bbad-4257087b71a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 02 01:33:28.919569 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:28.919529 4427 patch_prober.go:29] interesting pod/e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 02 01:33:28.919924 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:28.919881 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4" podUID="f8eafff6-528c-4e61-adff-8aed03dea982" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 02 01:33:29.245724 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:29.245642 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d" Nov 02 01:33:29.247862 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:29.247841 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d" Nov 02 01:33:29.252593 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:29.252573 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d" Nov 02 01:33:29.982739 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:29.982700 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4" Nov 02 01:33:29.984786 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:29.984762 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4" Nov 02 01:33:30.052677 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:30.050161 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4" Nov 02 01:33:31.649480 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:33:31.649437 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:33:42.423647 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:42.423570 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-37b78d-c9b5f9895-9j65q"] Nov 02 01:33:42.428034 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:42.428014 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-37b78d-c9b5f9895-9j65q" Nov 02 01:33:42.433818 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:42.433792 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-37b78d\"" Nov 02 01:33:42.434957 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:42.434922 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-37b78d-dockercfg-qdqd9\"" Nov 02 01:33:42.434957 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:42.434941 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-37b78d-667855b95-bhlr9"] Nov 02 01:33:42.440248 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:42.440214 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-37b78d-667855b95-bhlr9" Nov 02 01:33:42.440430 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:42.440410 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-37b78d-c9b5f9895-9j65q"] Nov 02 01:33:42.452850 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:42.452790 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-37b78d-667855b95-bhlr9"] Nov 02 01:33:42.516398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:42.515676 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvcz9\" (UniqueName: \"kubernetes.io/projected/30189b00-2aaa-4f2c-aa29-3f3df5d1dcc3-kube-api-access-hvcz9\") pod \"payload-pre-processing-e2e-api-b-37b78d-667855b95-bhlr9\" (UID: \"30189b00-2aaa-4f2c-aa29-3f3df5d1dcc3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-37b78d-667855b95-bhlr9" Nov 02 01:33:42.516398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:42.515869 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5207b6b-dc05-483f-add7-1b1a4ca76ea8-plugins-config-volume\") pod \"payload-processing-e2e-api-b-37b78d-c9b5f9895-9j65q\" (UID: \"a5207b6b-dc05-483f-add7-1b1a4ca76ea8\") " pod="openshift-ingress/payload-processing-e2e-api-b-37b78d-c9b5f9895-9j65q" Nov 02 01:33:42.516398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:42.515967 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30189b00-2aaa-4f2c-aa29-3f3df5d1dcc3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-37b78d-667855b95-bhlr9\" (UID: \"30189b00-2aaa-4f2c-aa29-3f3df5d1dcc3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-37b78d-667855b95-bhlr9" Nov 02 01:33:42.516398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:42.516094 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f5kv\" (UniqueName: \"kubernetes.io/projected/a5207b6b-dc05-483f-add7-1b1a4ca76ea8-kube-api-access-2f5kv\") pod \"payload-processing-e2e-api-b-37b78d-c9b5f9895-9j65q\" (UID: \"a5207b6b-dc05-483f-add7-1b1a4ca76ea8\") " pod="openshift-ingress/payload-processing-e2e-api-b-37b78d-c9b5f9895-9j65q" Nov 02 01:33:42.633399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:42.630591 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5207b6b-dc05-483f-add7-1b1a4ca76ea8-plugins-config-volume\") pod \"payload-processing-e2e-api-b-37b78d-c9b5f9895-9j65q\" (UID: \"a5207b6b-dc05-483f-add7-1b1a4ca76ea8\") " pod="openshift-ingress/payload-processing-e2e-api-b-37b78d-c9b5f9895-9j65q" Nov 02 01:33:42.633399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:42.630708 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30189b00-2aaa-4f2c-aa29-3f3df5d1dcc3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-37b78d-667855b95-bhlr9\" (UID: \"30189b00-2aaa-4f2c-aa29-3f3df5d1dcc3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-37b78d-667855b95-bhlr9" Nov 02 01:33:42.633399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:42.630902 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f5kv\" (UniqueName: \"kubernetes.io/projected/a5207b6b-dc05-483f-add7-1b1a4ca76ea8-kube-api-access-2f5kv\") pod \"payload-processing-e2e-api-b-37b78d-c9b5f9895-9j65q\" (UID: \"a5207b6b-dc05-483f-add7-1b1a4ca76ea8\") " pod="openshift-ingress/payload-processing-e2e-api-b-37b78d-c9b5f9895-9j65q" Nov 02 01:33:42.633399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:42.631069 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvcz9\" (UniqueName: \"kubernetes.io/projected/30189b00-2aaa-4f2c-aa29-3f3df5d1dcc3-kube-api-access-hvcz9\") pod \"payload-pre-processing-e2e-api-b-37b78d-667855b95-bhlr9\" (UID: \"30189b00-2aaa-4f2c-aa29-3f3df5d1dcc3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-37b78d-667855b95-bhlr9" Nov 02 01:33:42.648473 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:42.635635 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30189b00-2aaa-4f2c-aa29-3f3df5d1dcc3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-37b78d-667855b95-bhlr9\" (UID: \"30189b00-2aaa-4f2c-aa29-3f3df5d1dcc3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-37b78d-667855b95-bhlr9" Nov 02 01:33:42.648473 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:42.646691 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5207b6b-dc05-483f-add7-1b1a4ca76ea8-plugins-config-volume\") pod \"payload-processing-e2e-api-b-37b78d-c9b5f9895-9j65q\" (UID: \"a5207b6b-dc05-483f-add7-1b1a4ca76ea8\") " pod="openshift-ingress/payload-processing-e2e-api-b-37b78d-c9b5f9895-9j65q" Nov 02 01:33:42.717449 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:42.707641 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvcz9\" (UniqueName: \"kubernetes.io/projected/30189b00-2aaa-4f2c-aa29-3f3df5d1dcc3-kube-api-access-hvcz9\") pod \"payload-pre-processing-e2e-api-b-37b78d-667855b95-bhlr9\" (UID: \"30189b00-2aaa-4f2c-aa29-3f3df5d1dcc3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-37b78d-667855b95-bhlr9" Nov 02 01:33:42.759793 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:42.759643 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f5kv\" (UniqueName: \"kubernetes.io/projected/a5207b6b-dc05-483f-add7-1b1a4ca76ea8-kube-api-access-2f5kv\") pod \"payload-processing-e2e-api-b-37b78d-c9b5f9895-9j65q\" (UID: \"a5207b6b-dc05-483f-add7-1b1a4ca76ea8\") " pod="openshift-ingress/payload-processing-e2e-api-b-37b78d-c9b5f9895-9j65q" Nov 02 01:33:42.762460 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:42.761980 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-37b78d-667855b95-bhlr9" Nov 02 01:33:43.052751 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:43.052716 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-37b78d-c9b5f9895-9j65q" Nov 02 01:33:43.053053 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:43.053030 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-37b78d-667855b95-bhlr9"] Nov 02 01:33:43.053747 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:33:43.053713 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30189b00_2aaa_4f2c_aa29_3f3df5d1dcc3.slice/crio-b3f51adc42d4879aadbbbba4af63c1e8c34a11084a857c8b409c4057a5d23b43 WatchSource:0}: Error finding container b3f51adc42d4879aadbbbba4af63c1e8c34a11084a857c8b409c4057a5d23b43: Status 404 returned error can't find the container with id b3f51adc42d4879aadbbbba4af63c1e8c34a11084a857c8b409c4057a5d23b43 Nov 02 01:33:43.344311 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:43.344278 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-37b78d-c9b5f9895-9j65q"] Nov 02 01:33:43.344870 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:33:43.344840 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5207b6b_dc05_483f_add7_1b1a4ca76ea8.slice/crio-13d99794ca0a10f20e5b947eda7ec7c54b30818ef0bd9c791557ccf40bb2e636 WatchSource:0}: Error finding container 13d99794ca0a10f20e5b947eda7ec7c54b30818ef0bd9c791557ccf40bb2e636: Status 404 returned error can't find the container with id 13d99794ca0a10f20e5b947eda7ec7c54b30818ef0bd9c791557ccf40bb2e636 Nov 02 01:33:43.665907 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:43.665868 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-37b78d-c9b5f9895-9j65q" event={"ID":"a5207b6b-dc05-483f-add7-1b1a4ca76ea8","Type":"ContainerStarted","Data":"dff9103823d508991bb0a7653fc02b0f2dfb2b0cbe0e2c63600a567ed269248d"} Nov 02 01:33:43.666454 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:43.665950 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-37b78d-c9b5f9895-9j65q" Nov 02 01:33:43.666454 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:43.665975 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-37b78d-c9b5f9895-9j65q" event={"ID":"a5207b6b-dc05-483f-add7-1b1a4ca76ea8","Type":"ContainerStarted","Data":"13d99794ca0a10f20e5b947eda7ec7c54b30818ef0bd9c791557ccf40bb2e636"} Nov 02 01:33:43.667967 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:43.667935 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-37b78d-667855b95-bhlr9" event={"ID":"30189b00-2aaa-4f2c-aa29-3f3df5d1dcc3","Type":"ContainerStarted","Data":"f7b5b5804887e3cac1a2aa1241dba3978c4a8647df39c361c343b55d65cbf551"} Nov 02 01:33:43.668101 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:43.667988 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-37b78d-667855b95-bhlr9" event={"ID":"30189b00-2aaa-4f2c-aa29-3f3df5d1dcc3","Type":"ContainerStarted","Data":"b3f51adc42d4879aadbbbba4af63c1e8c34a11084a857c8b409c4057a5d23b43"} Nov 02 01:33:43.668101 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:43.668018 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-37b78d-667855b95-bhlr9" Nov 02 01:33:43.685107 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:43.685049 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-37b78d-c9b5f9895-9j65q" podStartSLOduration=1.685031168 podStartE2EDuration="1.685031168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:33:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:33:43.683021123 +0000 UTC m=+1128.304026334" watchObservedRunningTime="2026-11-02 01:33:43.685031168 +0000 UTC m=+1128.306036379" Nov 02 01:33:43.706171 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:43.706121 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-37b78d-667855b95-bhlr9" podStartSLOduration=1.706108365 podStartE2EDuration="1.706108365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:33:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:33:43.704968815 +0000 UTC m=+1128.325974050" watchObservedRunningTime="2026-11-02 01:33:43.706108365 +0000 UTC m=+1128.327113633" Nov 02 01:33:54.678829 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:54.678798 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-37b78d-c9b5f9895-9j65q" Nov 02 01:33:54.679244 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:54.678869 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-37b78d-667855b95-bhlr9" Nov 02 01:33:55.974670 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:55.974640 4427 scope.go:122] "RemoveContainer" containerID="0478aca3b5c933fc6d216abc2c10806a4f308c3183c001fc306f56c6b257f3e5" Nov 02 01:33:55.985356 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:33:55.985329 4427 scope.go:122] "RemoveContainer" containerID="8cf110a2d68b9d6d877d0a006337cde529667c6f5817791412fd15c54327fc33" Nov 02 01:34:01.583726 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:34:01.583687 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:34:15.718175 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:34:15.718138 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2ebe99-c55c4c955-fxcm6"] Nov 02 01:34:20.719110 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:34:20.719067 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-2ebe99-c55c4c955-fxcm6" podUID="79a5f4f6-4fad-4cfa-b5e7-562db4195d4a" containerName="payload-processing" containerID="cri-o://d7bc878094983d7327eca9233efcb948fdabbab2c77f2a05b3d42222665d5a81" gracePeriod=25 Nov 02 01:34:21.038718 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:34:21.038689 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2ebe99-c55c4c955-fxcm6" Nov 02 01:34:21.154470 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:34:21.154439 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79a5f4f6-4fad-4cfa-b5e7-562db4195d4a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79a5f4f6-4fad-4cfa-b5e7-562db4195d4a-plugins-config-volume\") pod \"79a5f4f6-4fad-4cfa-b5e7-562db4195d4a\" (UID: \"79a5f4f6-4fad-4cfa-b5e7-562db4195d4a\") " Nov 02 01:34:21.154689 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:34:21.154676 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79a5f4f6-4fad-4cfa-b5e7-562db4195d4a-kube-api-access-fnzv2\" (UniqueName: \"kubernetes.io/projected/79a5f4f6-4fad-4cfa-b5e7-562db4195d4a-kube-api-access-fnzv2\") pod \"79a5f4f6-4fad-4cfa-b5e7-562db4195d4a\" (UID: \"79a5f4f6-4fad-4cfa-b5e7-562db4195d4a\") " Nov 02 01:34:21.154910 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:34:21.154883 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79a5f4f6-4fad-4cfa-b5e7-562db4195d4a-plugins-config-volume" pod "79a5f4f6-4fad-4cfa-b5e7-562db4195d4a" (UID: "79a5f4f6-4fad-4cfa-b5e7-562db4195d4a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:34:21.155289 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:34:21.155263 4427 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79a5f4f6-4fad-4cfa-b5e7-562db4195d4a-plugins-config-volume\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:34:21.157043 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:34:21.157020 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79a5f4f6-4fad-4cfa-b5e7-562db4195d4a-kube-api-access-fnzv2" pod "79a5f4f6-4fad-4cfa-b5e7-562db4195d4a" (UID: "79a5f4f6-4fad-4cfa-b5e7-562db4195d4a"). InnerVolumeSpecName "kube-api-access-fnzv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:34:21.199233 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:34:21.199198 4427 generic.go:356] "Generic (PLEG): container finished" podID="79a5f4f6-4fad-4cfa-b5e7-562db4195d4a" containerID="d7bc878094983d7327eca9233efcb948fdabbab2c77f2a05b3d42222665d5a81" exitCode=0 Nov 02 01:34:21.199464 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:34:21.199265 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2ebe99-c55c4c955-fxcm6" Nov 02 01:34:21.199464 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:34:21.199294 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2ebe99-c55c4c955-fxcm6" event={"ID":"79a5f4f6-4fad-4cfa-b5e7-562db4195d4a","Type":"ContainerDied","Data":"d7bc878094983d7327eca9233efcb948fdabbab2c77f2a05b3d42222665d5a81"} Nov 02 01:34:21.199464 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:34:21.199364 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2ebe99-c55c4c955-fxcm6" event={"ID":"79a5f4f6-4fad-4cfa-b5e7-562db4195d4a","Type":"ContainerDied","Data":"aa830c846fc612934032dded73c4c73daadbdd53a031f906eb9e27584479cd0e"} Nov 02 01:34:21.199464 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:34:21.199412 4427 scope.go:122] "RemoveContainer" containerID="d7bc878094983d7327eca9233efcb948fdabbab2c77f2a05b3d42222665d5a81" Nov 02 01:34:21.211143 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:34:21.211123 4427 scope.go:122] "RemoveContainer" containerID="d7bc878094983d7327eca9233efcb948fdabbab2c77f2a05b3d42222665d5a81" Nov 02 01:34:21.211466 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:34:21.211445 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7bc878094983d7327eca9233efcb948fdabbab2c77f2a05b3d42222665d5a81\": container with ID starting with d7bc878094983d7327eca9233efcb948fdabbab2c77f2a05b3d42222665d5a81 not found: ID does not exist" containerID="d7bc878094983d7327eca9233efcb948fdabbab2c77f2a05b3d42222665d5a81" Nov 02 01:34:21.211606 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:34:21.211476 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7bc878094983d7327eca9233efcb948fdabbab2c77f2a05b3d42222665d5a81"} err="failed to get container status \"d7bc878094983d7327eca9233efcb948fdabbab2c77f2a05b3d42222665d5a81\": rpc error: code = NotFound desc = could not find container \"d7bc878094983d7327eca9233efcb948fdabbab2c77f2a05b3d42222665d5a81\": container with ID starting with d7bc878094983d7327eca9233efcb948fdabbab2c77f2a05b3d42222665d5a81 not found: ID does not exist" Nov 02 01:34:21.228286 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:34:21.228261 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2ebe99-c55c4c955-fxcm6"] Nov 02 01:34:21.232116 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:34:21.232083 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2ebe99-c55c4c955-fxcm6"] Nov 02 01:34:21.256549 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:34:21.256470 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnzv2\" (UniqueName: \"kubernetes.io/projected/79a5f4f6-4fad-4cfa-b5e7-562db4195d4a-kube-api-access-fnzv2\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:34:21.537705 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:34:21.537623 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79a5f4f6-4fad-4cfa-b5e7-562db4195d4a" path="/var/lib/kubelet/pods/79a5f4f6-4fad-4cfa-b5e7-562db4195d4a/volumes" Nov 02 01:34:31.584309 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:34:31.584265 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:34:32.037791 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:34:32.037738 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e26f5c63-8khb5"] Nov 02 01:34:32.044042 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:34:32.043174 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e26f5c63-8khb5"] Nov 02 01:34:33.535690 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:34:33.535659 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf7743f7-236d-4094-ae86-08fac541fd80" path="/var/lib/kubelet/pods/bf7743f7-236d-4094-ae86-08fac541fd80/volumes" Nov 02 01:34:49.078425 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:34:49.078389 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-677312-6fb455f55f-q8f9v"] Nov 02 01:34:49.079139 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:34:49.079123 4427 cpu_manager.go:414] "containerMap: removing container" podUID="79a5f4f6-4fad-4cfa-b5e7-562db4195d4a" containerName="payload-processing" Nov 02 01:34:49.079139 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:34:49.079140 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79a5f4f6-4fad-4cfa-b5e7-562db4195d4a" containerName="payload-processing" Nov 02 01:34:49.079261 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:34:49.079251 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="79a5f4f6-4fad-4cfa-b5e7-562db4195d4a" containerName="payload-processing" Nov 02 01:34:49.082844 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:34:49.082825 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-677312-6fb455f55f-q8f9v" Nov 02 01:34:49.089488 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:34:49.089469 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-677312-dockercfg-zj2tj\"" Nov 02 01:34:49.095001 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:34:49.094979 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-677312\"" Nov 02 01:34:49.099753 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:34:49.099730 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-677312-6fb455f55f-q8f9v"] Nov 02 01:34:49.131989 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:34:49.131942 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-677312-54c7456ddc-fvmf6"] Nov 02 01:34:49.136622 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:34:49.136602 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-677312-54c7456ddc-fvmf6" Nov 02 01:34:49.147850 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:34:49.147824 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-677312-54c7456ddc-fvmf6"] Nov 02 01:34:49.233693 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:34:49.230903 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/317a9ada-6f39-4dee-8945-9cc2dd1c18eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-677312-6fb455f55f-q8f9v\" (UID: \"317a9ada-6f39-4dee-8945-9cc2dd1c18eb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-677312-6fb455f55f-q8f9v" Nov 02 01:34:49.233693 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:34:49.231121 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d133c468-9c0c-4d15-8b0e-9292b27944f6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-677312-54c7456ddc-fvmf6\" (UID: \"d133c468-9c0c-4d15-8b0e-9292b27944f6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-677312-54c7456ddc-fvmf6" Nov 02 01:34:49.233693 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:34:49.231185 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9ltm\" (UniqueName: \"kubernetes.io/projected/d133c468-9c0c-4d15-8b0e-9292b27944f6-kube-api-access-j9ltm\") pod \"payload-processing-e2e-shared-a-w3-677312-54c7456ddc-fvmf6\" (UID: \"d133c468-9c0c-4d15-8b0e-9292b27944f6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-677312-54c7456ddc-fvmf6" Nov 02 01:34:49.233693 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:34:49.231268 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m46cj\" (UniqueName: \"kubernetes.io/projected/317a9ada-6f39-4dee-8945-9cc2dd1c18eb-kube-api-access-m46cj\") pod \"payload-pre-processing-e2e-shared-a-w3-677312-6fb455f55f-q8f9v\" (UID: \"317a9ada-6f39-4dee-8945-9cc2dd1c18eb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-677312-6fb455f55f-q8f9v" Nov 02 01:34:49.395597 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:34:49.391757 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d133c468-9c0c-4d15-8b0e-9292b27944f6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-677312-54c7456ddc-fvmf6\" (UID: \"d133c468-9c0c-4d15-8b0e-9292b27944f6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-677312-54c7456ddc-fvmf6" Nov 02 01:34:49.395597 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:34:49.391839 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9ltm\" (UniqueName: \"kubernetes.io/projected/d133c468-9c0c-4d15-8b0e-9292b27944f6-kube-api-access-j9ltm\") pod \"payload-processing-e2e-shared-a-w3-677312-54c7456ddc-fvmf6\" (UID: \"d133c468-9c0c-4d15-8b0e-9292b27944f6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-677312-54c7456ddc-fvmf6" Nov 02 01:34:49.395597 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:34:49.391913 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m46cj\" (UniqueName: \"kubernetes.io/projected/317a9ada-6f39-4dee-8945-9cc2dd1c18eb-kube-api-access-m46cj\") pod \"payload-pre-processing-e2e-shared-a-w3-677312-6fb455f55f-q8f9v\" (UID: \"317a9ada-6f39-4dee-8945-9cc2dd1c18eb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-677312-6fb455f55f-q8f9v" Nov 02 01:34:49.395597 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:34:49.392150 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/317a9ada-6f39-4dee-8945-9cc2dd1c18eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-677312-6fb455f55f-q8f9v\" (UID: \"317a9ada-6f39-4dee-8945-9cc2dd1c18eb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-677312-6fb455f55f-q8f9v" Nov 02 01:34:49.395597 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:34:49.393363 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/317a9ada-6f39-4dee-8945-9cc2dd1c18eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-677312-6fb455f55f-q8f9v\" (UID: \"317a9ada-6f39-4dee-8945-9cc2dd1c18eb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-677312-6fb455f55f-q8f9v" Nov 02 01:34:49.395597 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:34:49.394147 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d133c468-9c0c-4d15-8b0e-9292b27944f6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-677312-54c7456ddc-fvmf6\" (UID: \"d133c468-9c0c-4d15-8b0e-9292b27944f6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-677312-54c7456ddc-fvmf6" Nov 02 01:34:49.410460 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:34:49.410358 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9ltm\" (UniqueName: \"kubernetes.io/projected/d133c468-9c0c-4d15-8b0e-9292b27944f6-kube-api-access-j9ltm\") pod \"payload-processing-e2e-shared-a-w3-677312-54c7456ddc-fvmf6\" (UID: \"d133c468-9c0c-4d15-8b0e-9292b27944f6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-677312-54c7456ddc-fvmf6" Nov 02 01:34:49.414075 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:34:49.414017 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m46cj\" (UniqueName: \"kubernetes.io/projected/317a9ada-6f39-4dee-8945-9cc2dd1c18eb-kube-api-access-m46cj\") pod \"payload-pre-processing-e2e-shared-a-w3-677312-6fb455f55f-q8f9v\" (UID: \"317a9ada-6f39-4dee-8945-9cc2dd1c18eb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-677312-6fb455f55f-q8f9v" Nov 02 01:34:49.463298 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:34:49.463201 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-677312-54c7456ddc-fvmf6" Nov 02 01:34:49.698190 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:34:49.698085 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-677312-6fb455f55f-q8f9v" Nov 02 01:34:49.758707 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:34:49.758222 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-677312-54c7456ddc-fvmf6"] Nov 02 01:34:50.002217 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:34:50.001890 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod317a9ada_6f39_4dee_8945_9cc2dd1c18eb.slice/crio-d5601a6b2046bd179407adb5cb7df8c0fb1195b6d0ec6abc65874185955ad661 WatchSource:0}: Error finding container d5601a6b2046bd179407adb5cb7df8c0fb1195b6d0ec6abc65874185955ad661: Status 404 returned error can't find the container with id d5601a6b2046bd179407adb5cb7df8c0fb1195b6d0ec6abc65874185955ad661 Nov 02 01:34:50.003017 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:34:50.002787 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-677312-6fb455f55f-q8f9v"] Nov 02 01:34:50.491264 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:34:50.491224 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-677312-54c7456ddc-fvmf6" event={"ID":"d133c468-9c0c-4d15-8b0e-9292b27944f6","Type":"ContainerStarted","Data":"c4616ecde2911c4639b4be90570a5e27c0bf897d2ba101d2fef378d4cd0e9be5"} Nov 02 01:34:50.491812 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:34:50.491300 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-677312-54c7456ddc-fvmf6" Nov 02 01:34:50.491812 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:34:50.491329 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-677312-54c7456ddc-fvmf6" event={"ID":"d133c468-9c0c-4d15-8b0e-9292b27944f6","Type":"ContainerStarted","Data":"5a776d891dfb88d01c8c4f293a8e369e67b4d8f16954c929666cf6ab7489970e"} Nov 02 01:34:50.492934 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:34:50.492908 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-677312-6fb455f55f-q8f9v" event={"ID":"317a9ada-6f39-4dee-8945-9cc2dd1c18eb","Type":"ContainerStarted","Data":"95c6d74cf33ff7289e109f010c4a398e8733ad63840ceff059fc44061b75999c"} Nov 02 01:34:50.493068 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:34:50.492953 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-677312-6fb455f55f-q8f9v" Nov 02 01:34:50.493068 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:34:50.492970 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-677312-6fb455f55f-q8f9v" event={"ID":"317a9ada-6f39-4dee-8945-9cc2dd1c18eb","Type":"ContainerStarted","Data":"d5601a6b2046bd179407adb5cb7df8c0fb1195b6d0ec6abc65874185955ad661"} Nov 02 01:34:50.512810 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:34:50.512755 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-677312-54c7456ddc-fvmf6" podStartSLOduration=1.5127431709999999 podStartE2EDuration="1.512743171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:34:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:34:50.509517007 +0000 UTC m=+1195.130522218" watchObservedRunningTime="2026-11-02 01:34:50.512743171 +0000 UTC m=+1195.133748379" Nov 02 01:34:50.528643 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:34:50.528593 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-677312-6fb455f55f-q8f9v" podStartSLOduration=1.5285819379999999 podStartE2EDuration="1.528581938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:34:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:34:50.527448506 +0000 UTC m=+1195.148453720" watchObservedRunningTime="2026-11-02 01:34:50.528581938 +0000 UTC m=+1195.149587148" Nov 02 01:34:54.097849 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:34:54.097813 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03a8ca0d-z4v8m"] Nov 02 01:34:54.112011 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:34:54.111979 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03a8ca0d-z4v8m"] Nov 02 01:34:55.537610 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:34:55.537581 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf676aff-c089-43c6-95de-45c72e2c22ab" path="/var/lib/kubelet/pods/bf676aff-c089-43c6-95de-45c72e2c22ab/volumes" Nov 02 01:34:57.133101 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:34:57.133064 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-37b78d-c9b5f9895-9j65q"] Nov 02 01:34:57.145310 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:34:57.145275 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-37b78d-667855b95-bhlr9"] Nov 02 01:35:01.136275 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:01.136234 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l"] Nov 02 01:35:01.160631 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:01.160597 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l"] Nov 02 01:35:01.160867 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:01.160760 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l" Nov 02 01:35:01.165021 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:01.164997 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-0bbeda-openshift-default-dockercfg-8m89f\"" Nov 02 01:35:01.243719 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:01.243687 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0-istio-envoy\") pod \"e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l\" (UID: \"e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0\") " pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l" Nov 02 01:35:01.243900 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:01.243731 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0-workload-socket\") pod \"e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l\" (UID: \"e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0\") " pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l" Nov 02 01:35:01.243900 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:01.243844 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0-credential-socket\") pod \"e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l\" (UID: \"e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0\") " pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l" Nov 02 01:35:01.244001 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:01.243976 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0-istio-token\") pod \"e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l\" (UID: \"e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0\") " pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l" Nov 02 01:35:01.244045 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:01.244010 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0-istiod-ca-cert\") pod \"e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l\" (UID: \"e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0\") " pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l" Nov 02 01:35:01.244135 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:01.244121 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0-istio-data\") pod \"e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l\" (UID: \"e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0\") " pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l" Nov 02 01:35:01.244170 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:01.244154 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0-workload-certs\") pod \"e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l\" (UID: \"e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0\") " pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l" Nov 02 01:35:01.244238 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:01.244217 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zchs\" (UniqueName: \"kubernetes.io/projected/e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0-kube-api-access-7zchs\") pod \"e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l\" (UID: \"e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0\") " pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l" Nov 02 01:35:01.244470 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:01.244336 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0-istio-podinfo\") pod \"e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l\" (UID: \"e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0\") " pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l" Nov 02 01:35:01.357407 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:01.356695 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0-istio-envoy\") pod \"e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l\" (UID: \"e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0\") " pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l" Nov 02 01:35:01.357407 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:01.356779 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0-workload-socket\") pod \"e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l\" (UID: \"e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0\") " pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l" Nov 02 01:35:01.357407 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:01.356836 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0-credential-socket\") pod \"e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l\" (UID: \"e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0\") " pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l" Nov 02 01:35:01.357407 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:01.356923 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0-istio-token\") pod \"e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l\" (UID: \"e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0\") " pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l" Nov 02 01:35:01.357407 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:01.356960 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0-istiod-ca-cert\") pod \"e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l\" (UID: \"e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0\") " pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l" Nov 02 01:35:01.357407 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:01.357049 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0-istio-data\") pod \"e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l\" (UID: \"e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0\") " pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l" Nov 02 01:35:01.357407 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:01.357096 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0-workload-certs\") pod \"e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l\" (UID: \"e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0\") " pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l" Nov 02 01:35:01.357407 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:01.357140 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zchs\" (UniqueName: \"kubernetes.io/projected/e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0-kube-api-access-7zchs\") pod \"e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l\" (UID: \"e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0\") " pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l" Nov 02 01:35:01.357407 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:01.357225 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0-istio-podinfo\") pod \"e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l\" (UID: \"e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0\") " pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l" Nov 02 01:35:01.360971 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:01.358192 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0-istio-data\") pod \"e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l\" (UID: \"e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0\") " pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l" Nov 02 01:35:01.360971 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:01.358845 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0-istiod-ca-cert\") pod \"e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l\" (UID: \"e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0\") " pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l" Nov 02 01:35:01.360971 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:01.359095 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0-workload-certs\") pod \"e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l\" (UID: \"e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0\") " pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l" Nov 02 01:35:01.360971 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:01.359324 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0-workload-socket\") pod \"e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l\" (UID: \"e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0\") " pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l" Nov 02 01:35:01.370395 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:01.365802 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0-credential-socket\") pod \"e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l\" (UID: \"e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0\") " pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l" Nov 02 01:35:01.406614 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:01.401348 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zchs\" (UniqueName: \"kubernetes.io/projected/e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0-kube-api-access-7zchs\") pod \"e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l\" (UID: \"e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0\") " pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l" Nov 02 01:35:01.407407 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:01.407359 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0-istio-podinfo\") pod \"e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l\" (UID: \"e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0\") " pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l" Nov 02 01:35:01.432398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:01.426323 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0-istio-envoy\") pod \"e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l\" (UID: \"e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0\") " pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l" Nov 02 01:35:01.432398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:01.430062 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0-istio-token\") pod \"e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l\" (UID: \"e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0\") " pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l" Nov 02 01:35:01.485972 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:01.484689 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l" Nov 02 01:35:01.582435 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:01.581549 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-677312-6fb455f55f-q8f9v" Nov 02 01:35:01.582435 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:01.581657 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-677312-54c7456ddc-fvmf6" Nov 02 01:35:01.707630 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:35:01.707523 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:35:02.037153 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:02.037120 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l"] Nov 02 01:35:02.039415 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:35:02.039349 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode32db4d5_2c23_4d7a_8a3d_ec7866cb35a0.slice/crio-a8e28c7f3bf765c1c5bfe29c6317dca8f4b80825096b35d99f7bc2063e1fb19c WatchSource:0}: Error finding container a8e28c7f3bf765c1c5bfe29c6317dca8f4b80825096b35d99f7bc2063e1fb19c: Status 404 returned error can't find the container with id a8e28c7f3bf765c1c5bfe29c6317dca8f4b80825096b35d99f7bc2063e1fb19c Nov 02 01:35:02.041835 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:02.041799 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:35:02.041923 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:02.041885 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:35:02.041923 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:02.041914 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:35:02.134025 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:02.133984 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-37b78d-c9b5f9895-9j65q" podUID="a5207b6b-dc05-483f-add7-1b1a4ca76ea8" containerName="payload-processing" containerID="cri-o://dff9103823d508991bb0a7653fc02b0f2dfb2b0cbe0e2c63600a567ed269248d" gracePeriod=25 Nov 02 01:35:02.150430 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:02.148446 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-37b78d-667855b95-bhlr9" podUID="30189b00-2aaa-4f2c-aa29-3f3df5d1dcc3" containerName="payload-pre-processing" containerID="cri-o://f7b5b5804887e3cac1a2aa1241dba3978c4a8647df39c361c343b55d65cbf551" gracePeriod=25 Nov 02 01:35:02.577646 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:02.577198 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-37b78d-c9b5f9895-9j65q" Nov 02 01:35:02.726094 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:02.725175 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-37b78d-667855b95-bhlr9" Nov 02 01:35:02.735527 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:02.735495 4427 generic.go:356] "Generic (PLEG): container finished" podID="a5207b6b-dc05-483f-add7-1b1a4ca76ea8" containerID="dff9103823d508991bb0a7653fc02b0f2dfb2b0cbe0e2c63600a567ed269248d" exitCode=0 Nov 02 01:35:02.735850 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:02.735824 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-37b78d-c9b5f9895-9j65q" event={"ID":"a5207b6b-dc05-483f-add7-1b1a4ca76ea8","Type":"ContainerDied","Data":"dff9103823d508991bb0a7653fc02b0f2dfb2b0cbe0e2c63600a567ed269248d"} Nov 02 01:35:02.736065 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:02.736045 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-37b78d-c9b5f9895-9j65q" event={"ID":"a5207b6b-dc05-483f-add7-1b1a4ca76ea8","Type":"ContainerDied","Data":"13d99794ca0a10f20e5b947eda7ec7c54b30818ef0bd9c791557ccf40bb2e636"} Nov 02 01:35:02.736194 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:02.736179 4427 scope.go:122] "RemoveContainer" containerID="dff9103823d508991bb0a7653fc02b0f2dfb2b0cbe0e2c63600a567ed269248d" Nov 02 01:35:02.736480 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:02.736461 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-37b78d-c9b5f9895-9j65q" Nov 02 01:35:02.741853 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:02.741827 4427 generic.go:356] "Generic (PLEG): container finished" podID="30189b00-2aaa-4f2c-aa29-3f3df5d1dcc3" containerID="f7b5b5804887e3cac1a2aa1241dba3978c4a8647df39c361c343b55d65cbf551" exitCode=0 Nov 02 01:35:02.742046 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:02.742026 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-37b78d-667855b95-bhlr9" event={"ID":"30189b00-2aaa-4f2c-aa29-3f3df5d1dcc3","Type":"ContainerDied","Data":"f7b5b5804887e3cac1a2aa1241dba3978c4a8647df39c361c343b55d65cbf551"} Nov 02 01:35:02.742204 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:02.742187 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-37b78d-667855b95-bhlr9" event={"ID":"30189b00-2aaa-4f2c-aa29-3f3df5d1dcc3","Type":"ContainerDied","Data":"b3f51adc42d4879aadbbbba4af63c1e8c34a11084a857c8b409c4057a5d23b43"} Nov 02 01:35:02.742392 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:02.742358 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-37b78d-667855b95-bhlr9" Nov 02 01:35:02.746561 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:02.746537 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5207b6b-dc05-483f-add7-1b1a4ca76ea8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5207b6b-dc05-483f-add7-1b1a4ca76ea8-plugins-config-volume\") pod \"a5207b6b-dc05-483f-add7-1b1a4ca76ea8\" (UID: \"a5207b6b-dc05-483f-add7-1b1a4ca76ea8\") " Nov 02 01:35:02.747723 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:02.747217 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5207b6b-dc05-483f-add7-1b1a4ca76ea8-plugins-config-volume" pod "a5207b6b-dc05-483f-add7-1b1a4ca76ea8" (UID: "a5207b6b-dc05-483f-add7-1b1a4ca76ea8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:35:02.747880 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:02.747863 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5207b6b-dc05-483f-add7-1b1a4ca76ea8-kube-api-access-2f5kv\" (UniqueName: \"kubernetes.io/projected/a5207b6b-dc05-483f-add7-1b1a4ca76ea8-kube-api-access-2f5kv\") pod \"a5207b6b-dc05-483f-add7-1b1a4ca76ea8\" (UID: \"a5207b6b-dc05-483f-add7-1b1a4ca76ea8\") " Nov 02 01:35:02.750918 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:02.750892 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l" event={"ID":"e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0","Type":"ContainerStarted","Data":"db7574e374b5d799907ecd4f3b953b8d7c2afccfa515149ab1a326ea05191cae"} Nov 02 01:35:02.751127 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:02.751110 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l" event={"ID":"e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0","Type":"ContainerStarted","Data":"a8e28c7f3bf765c1c5bfe29c6317dca8f4b80825096b35d99f7bc2063e1fb19c"} Nov 02 01:35:02.752204 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:02.752184 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5207b6b-dc05-483f-add7-1b1a4ca76ea8-kube-api-access-2f5kv" pod "a5207b6b-dc05-483f-add7-1b1a4ca76ea8" (UID: "a5207b6b-dc05-483f-add7-1b1a4ca76ea8"). InnerVolumeSpecName "kube-api-access-2f5kv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:35:02.753817 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:02.749362 4427 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5207b6b-dc05-483f-add7-1b1a4ca76ea8-plugins-config-volume\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:35:02.758350 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:02.758330 4427 scope.go:122] "RemoveContainer" containerID="dff9103823d508991bb0a7653fc02b0f2dfb2b0cbe0e2c63600a567ed269248d" Nov 02 01:35:02.758806 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:35:02.758773 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dff9103823d508991bb0a7653fc02b0f2dfb2b0cbe0e2c63600a567ed269248d\": container with ID starting with dff9103823d508991bb0a7653fc02b0f2dfb2b0cbe0e2c63600a567ed269248d not found: ID does not exist" containerID="dff9103823d508991bb0a7653fc02b0f2dfb2b0cbe0e2c63600a567ed269248d" Nov 02 01:35:02.758897 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:02.758806 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dff9103823d508991bb0a7653fc02b0f2dfb2b0cbe0e2c63600a567ed269248d"} err="failed to get container status \"dff9103823d508991bb0a7653fc02b0f2dfb2b0cbe0e2c63600a567ed269248d\": rpc error: code = NotFound desc = could not find container \"dff9103823d508991bb0a7653fc02b0f2dfb2b0cbe0e2c63600a567ed269248d\": container with ID starting with dff9103823d508991bb0a7653fc02b0f2dfb2b0cbe0e2c63600a567ed269248d not found: ID does not exist" Nov 02 01:35:02.758897 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:02.758825 4427 scope.go:122] "RemoveContainer" containerID="f7b5b5804887e3cac1a2aa1241dba3978c4a8647df39c361c343b55d65cbf551" Nov 02 01:35:02.782718 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:02.776754 4427 scope.go:122] "RemoveContainer" containerID="f7b5b5804887e3cac1a2aa1241dba3978c4a8647df39c361c343b55d65cbf551" Nov 02 01:35:02.782718 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:35:02.777327 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7b5b5804887e3cac1a2aa1241dba3978c4a8647df39c361c343b55d65cbf551\": container with ID starting with f7b5b5804887e3cac1a2aa1241dba3978c4a8647df39c361c343b55d65cbf551 not found: ID does not exist" containerID="f7b5b5804887e3cac1a2aa1241dba3978c4a8647df39c361c343b55d65cbf551" Nov 02 01:35:02.782718 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:02.777360 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7b5b5804887e3cac1a2aa1241dba3978c4a8647df39c361c343b55d65cbf551"} err="failed to get container status \"f7b5b5804887e3cac1a2aa1241dba3978c4a8647df39c361c343b55d65cbf551\": rpc error: code = NotFound desc = could not find container \"f7b5b5804887e3cac1a2aa1241dba3978c4a8647df39c361c343b55d65cbf551\": container with ID starting with f7b5b5804887e3cac1a2aa1241dba3978c4a8647df39c361c343b55d65cbf551 not found: ID does not exist" Nov 02 01:35:02.793841 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:02.793773 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l" podStartSLOduration=1.7937500339999999 podStartE2EDuration="1.793750034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:35:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:35:02.78694472 +0000 UTC m=+1207.407949932" watchObservedRunningTime="2026-11-02 01:35:02.793750034 +0000 UTC m=+1207.414755246" Nov 02 01:35:02.855426 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:02.855282 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30189b00-2aaa-4f2c-aa29-3f3df5d1dcc3-kube-api-access-hvcz9\" (UniqueName: \"kubernetes.io/projected/30189b00-2aaa-4f2c-aa29-3f3df5d1dcc3-kube-api-access-hvcz9\") pod \"30189b00-2aaa-4f2c-aa29-3f3df5d1dcc3\" (UID: \"30189b00-2aaa-4f2c-aa29-3f3df5d1dcc3\") " Nov 02 01:35:02.855426 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:02.855368 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30189b00-2aaa-4f2c-aa29-3f3df5d1dcc3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30189b00-2aaa-4f2c-aa29-3f3df5d1dcc3-plugins-config-volume\") pod \"30189b00-2aaa-4f2c-aa29-3f3df5d1dcc3\" (UID: \"30189b00-2aaa-4f2c-aa29-3f3df5d1dcc3\") " Nov 02 01:35:02.855850 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:02.855817 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30189b00-2aaa-4f2c-aa29-3f3df5d1dcc3-plugins-config-volume" pod "30189b00-2aaa-4f2c-aa29-3f3df5d1dcc3" (UID: "30189b00-2aaa-4f2c-aa29-3f3df5d1dcc3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:35:02.857339 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:02.856350 4427 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30189b00-2aaa-4f2c-aa29-3f3df5d1dcc3-plugins-config-volume\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:35:02.857339 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:02.856393 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f5kv\" (UniqueName: \"kubernetes.io/projected/a5207b6b-dc05-483f-add7-1b1a4ca76ea8-kube-api-access-2f5kv\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:35:02.859012 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:02.858978 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30189b00-2aaa-4f2c-aa29-3f3df5d1dcc3-kube-api-access-hvcz9" pod "30189b00-2aaa-4f2c-aa29-3f3df5d1dcc3" (UID: "30189b00-2aaa-4f2c-aa29-3f3df5d1dcc3"). InnerVolumeSpecName "kube-api-access-hvcz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:35:02.960881 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:02.958848 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvcz9\" (UniqueName: \"kubernetes.io/projected/30189b00-2aaa-4f2c-aa29-3f3df5d1dcc3-kube-api-access-hvcz9\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:35:03.134331 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:03.134261 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-37b78d-c9b5f9895-9j65q"] Nov 02 01:35:03.150557 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:03.147430 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-37b78d-c9b5f9895-9j65q"] Nov 02 01:35:03.154849 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:03.154816 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-37b78d-667855b95-bhlr9"] Nov 02 01:35:03.162848 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:03.162769 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-37b78d-667855b95-bhlr9"] Nov 02 01:35:03.489309 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:03.489223 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l" Nov 02 01:35:03.495658 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:03.495634 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l" Nov 02 01:35:03.540447 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:03.540336 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30189b00-2aaa-4f2c-aa29-3f3df5d1dcc3" path="/var/lib/kubelet/pods/30189b00-2aaa-4f2c-aa29-3f3df5d1dcc3/volumes" Nov 02 01:35:03.541142 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:03.541121 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5207b6b-dc05-483f-add7-1b1a4ca76ea8" path="/var/lib/kubelet/pods/a5207b6b-dc05-483f-add7-1b1a4ca76ea8/volumes" Nov 02 01:35:03.762575 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:03.762479 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l" Nov 02 01:35:03.763461 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:03.763439 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l" Nov 02 01:35:13.156235 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:13.156146 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-50537f4f-67d6cc6b87-kqzbs"] Nov 02 01:35:14.019603 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:14.019553 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp"] Nov 02 01:35:14.020685 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:14.020655 4427 cpu_manager.go:414] "containerMap: removing container" podUID="a5207b6b-dc05-483f-add7-1b1a4ca76ea8" containerName="payload-processing" Nov 02 01:35:14.020685 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:14.020685 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5207b6b-dc05-483f-add7-1b1a4ca76ea8" containerName="payload-processing" Nov 02 01:35:14.020896 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:14.020783 4427 cpu_manager.go:414] "containerMap: removing container" podUID="30189b00-2aaa-4f2c-aa29-3f3df5d1dcc3" containerName="payload-pre-processing" Nov 02 01:35:14.020896 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:14.020791 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30189b00-2aaa-4f2c-aa29-3f3df5d1dcc3" containerName="payload-pre-processing" Nov 02 01:35:14.021015 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:14.020943 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="30189b00-2aaa-4f2c-aa29-3f3df5d1dcc3" containerName="payload-pre-processing" Nov 02 01:35:14.021015 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:14.020962 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5207b6b-dc05-483f-add7-1b1a4ca76ea8" containerName="payload-processing" Nov 02 01:35:14.025452 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:14.025427 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp" Nov 02 01:35:14.037693 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:14.036497 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-0e06f2-openshift-default-dockercfg-j7ckn\"" Nov 02 01:35:14.044025 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:14.043980 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp"] Nov 02 01:35:14.127402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:14.121493 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f9b0669-291d-43fa-9511-d74948b03684-istio-token\") pod \"e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp\" (UID: \"8f9b0669-291d-43fa-9511-d74948b03684\") " pod="openshift-ingress/e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp" Nov 02 01:35:14.127402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:14.121590 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f9b0669-291d-43fa-9511-d74948b03684-istio-podinfo\") pod \"e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp\" (UID: \"8f9b0669-291d-43fa-9511-d74948b03684\") " pod="openshift-ingress/e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp" Nov 02 01:35:14.127402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:14.121646 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f9b0669-291d-43fa-9511-d74948b03684-istiod-ca-cert\") pod \"e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp\" (UID: \"8f9b0669-291d-43fa-9511-d74948b03684\") " pod="openshift-ingress/e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp" Nov 02 01:35:14.127402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:14.121675 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f9b0669-291d-43fa-9511-d74948b03684-workload-certs\") pod \"e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp\" (UID: \"8f9b0669-291d-43fa-9511-d74948b03684\") " pod="openshift-ingress/e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp" Nov 02 01:35:14.127402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:14.121755 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f9b0669-291d-43fa-9511-d74948b03684-istio-envoy\") pod \"e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp\" (UID: \"8f9b0669-291d-43fa-9511-d74948b03684\") " pod="openshift-ingress/e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp" Nov 02 01:35:14.127402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:14.121876 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f9b0669-291d-43fa-9511-d74948b03684-istio-data\") pod \"e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp\" (UID: \"8f9b0669-291d-43fa-9511-d74948b03684\") " pod="openshift-ingress/e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp" Nov 02 01:35:14.127402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:14.121904 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f9b0669-291d-43fa-9511-d74948b03684-credential-socket\") pod \"e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp\" (UID: \"8f9b0669-291d-43fa-9511-d74948b03684\") " pod="openshift-ingress/e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp" Nov 02 01:35:14.127402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:14.122054 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f9b0669-291d-43fa-9511-d74948b03684-workload-socket\") pod \"e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp\" (UID: \"8f9b0669-291d-43fa-9511-d74948b03684\") " pod="openshift-ingress/e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp" Nov 02 01:35:14.127402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:14.122103 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg7v2\" (UniqueName: \"kubernetes.io/projected/8f9b0669-291d-43fa-9511-d74948b03684-kube-api-access-qg7v2\") pod \"e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp\" (UID: \"8f9b0669-291d-43fa-9511-d74948b03684\") " pod="openshift-ingress/e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp" Nov 02 01:35:14.245402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:14.242732 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f9b0669-291d-43fa-9511-d74948b03684-istio-token\") pod \"e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp\" (UID: \"8f9b0669-291d-43fa-9511-d74948b03684\") " pod="openshift-ingress/e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp" Nov 02 01:35:14.245402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:14.242823 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f9b0669-291d-43fa-9511-d74948b03684-istio-podinfo\") pod \"e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp\" (UID: \"8f9b0669-291d-43fa-9511-d74948b03684\") " pod="openshift-ingress/e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp" Nov 02 01:35:14.245402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:14.242900 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f9b0669-291d-43fa-9511-d74948b03684-istiod-ca-cert\") pod \"e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp\" (UID: \"8f9b0669-291d-43fa-9511-d74948b03684\") " pod="openshift-ingress/e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp" Nov 02 01:35:14.245402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:14.242932 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f9b0669-291d-43fa-9511-d74948b03684-workload-certs\") pod \"e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp\" (UID: \"8f9b0669-291d-43fa-9511-d74948b03684\") " pod="openshift-ingress/e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp" Nov 02 01:35:14.245402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:14.243010 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f9b0669-291d-43fa-9511-d74948b03684-istio-envoy\") pod \"e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp\" (UID: \"8f9b0669-291d-43fa-9511-d74948b03684\") " pod="openshift-ingress/e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp" Nov 02 01:35:14.245402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:14.243138 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f9b0669-291d-43fa-9511-d74948b03684-istio-data\") pod \"e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp\" (UID: \"8f9b0669-291d-43fa-9511-d74948b03684\") " pod="openshift-ingress/e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp" Nov 02 01:35:14.245402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:14.243169 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f9b0669-291d-43fa-9511-d74948b03684-credential-socket\") pod \"e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp\" (UID: \"8f9b0669-291d-43fa-9511-d74948b03684\") " pod="openshift-ingress/e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp" Nov 02 01:35:14.245402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:14.243326 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f9b0669-291d-43fa-9511-d74948b03684-workload-socket\") pod \"e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp\" (UID: \"8f9b0669-291d-43fa-9511-d74948b03684\") " pod="openshift-ingress/e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp" Nov 02 01:35:14.245402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:14.243391 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg7v2\" (UniqueName: \"kubernetes.io/projected/8f9b0669-291d-43fa-9511-d74948b03684-kube-api-access-qg7v2\") pod \"e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp\" (UID: \"8f9b0669-291d-43fa-9511-d74948b03684\") " pod="openshift-ingress/e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp" Nov 02 01:35:14.260045 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:14.259554 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f9b0669-291d-43fa-9511-d74948b03684-credential-socket\") pod \"e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp\" (UID: \"8f9b0669-291d-43fa-9511-d74948b03684\") " pod="openshift-ingress/e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp" Nov 02 01:35:14.278622 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:14.260559 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f9b0669-291d-43fa-9511-d74948b03684-istiod-ca-cert\") pod \"e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp\" (UID: \"8f9b0669-291d-43fa-9511-d74948b03684\") " pod="openshift-ingress/e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp" Nov 02 01:35:14.278622 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:14.260664 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f9b0669-291d-43fa-9511-d74948b03684-workload-certs\") pod \"e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp\" (UID: \"8f9b0669-291d-43fa-9511-d74948b03684\") " pod="openshift-ingress/e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp" Nov 02 01:35:14.278622 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:14.261029 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f9b0669-291d-43fa-9511-d74948b03684-workload-socket\") pod \"e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp\" (UID: \"8f9b0669-291d-43fa-9511-d74948b03684\") " pod="openshift-ingress/e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp" Nov 02 01:35:14.279927 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:14.279899 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f9b0669-291d-43fa-9511-d74948b03684-istio-data\") pod \"e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp\" (UID: \"8f9b0669-291d-43fa-9511-d74948b03684\") " pod="openshift-ingress/e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp" Nov 02 01:35:14.297532 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:14.297495 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg7v2\" (UniqueName: \"kubernetes.io/projected/8f9b0669-291d-43fa-9511-d74948b03684-kube-api-access-qg7v2\") pod \"e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp\" (UID: \"8f9b0669-291d-43fa-9511-d74948b03684\") " pod="openshift-ingress/e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp" Nov 02 01:35:14.312926 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:14.312848 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f9b0669-291d-43fa-9511-d74948b03684-istio-envoy\") pod \"e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp\" (UID: \"8f9b0669-291d-43fa-9511-d74948b03684\") " pod="openshift-ingress/e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp" Nov 02 01:35:14.350257 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:14.348621 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f9b0669-291d-43fa-9511-d74948b03684-istio-podinfo\") pod \"e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp\" (UID: \"8f9b0669-291d-43fa-9511-d74948b03684\") " pod="openshift-ingress/e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp" Nov 02 01:35:14.350257 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:14.349069 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f9b0669-291d-43fa-9511-d74948b03684-istio-token\") pod \"e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp\" (UID: \"8f9b0669-291d-43fa-9511-d74948b03684\") " pod="openshift-ingress/e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp" Nov 02 01:35:14.364516 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:14.364229 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp" Nov 02 01:35:14.869545 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:14.869509 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp"] Nov 02 01:35:14.871152 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:35:14.871121 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f9b0669_291d_43fa_9511_d74948b03684.slice/crio-f8f08b2d8126733ec5e6122b23ffd39f98f510175c65804eb2d5de0f95ae0fab WatchSource:0}: Error finding container f8f08b2d8126733ec5e6122b23ffd39f98f510175c65804eb2d5de0f95ae0fab: Status 404 returned error can't find the container with id f8f08b2d8126733ec5e6122b23ffd39f98f510175c65804eb2d5de0f95ae0fab Nov 02 01:35:14.873948 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:14.873915 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:35:14.874072 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:14.874002 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:35:14.874072 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:14.874031 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:35:14.929900 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:14.929807 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp" event={"ID":"8f9b0669-291d-43fa-9511-d74948b03684","Type":"ContainerStarted","Data":"f8f08b2d8126733ec5e6122b23ffd39f98f510175c65804eb2d5de0f95ae0fab"} Nov 02 01:35:16.001405 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:15.999238 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp" event={"ID":"8f9b0669-291d-43fa-9511-d74948b03684","Type":"ContainerStarted","Data":"1a94ae972be836c0ad68ee977315577de6860497757ed3ef245783a084447f50"} Nov 02 01:35:16.040361 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:16.040292 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp" podStartSLOduration=3.040267151 podStartE2EDuration="3.040267151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:35:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:35:16.02862826 +0000 UTC m=+1220.649633473" watchObservedRunningTime="2026-11-02 01:35:16.040267151 +0000 UTC m=+1220.661272365" Nov 02 01:35:16.365273 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:16.365227 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp" Nov 02 01:35:16.373213 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:16.373183 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp" Nov 02 01:35:17.011012 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:17.010981 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp" Nov 02 01:35:17.012640 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:17.012619 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp" Nov 02 01:35:18.157350 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:18.157308 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-50537f4f-67d6cc6b87-kqzbs" podUID="d7877d44-4cab-4c19-a7f1-d761b2be0bb6" containerName="payload-processing" containerID="cri-o://e0c0a7f318355c0d5c099cfdea9e8e9c2f2e237a8ac65493a667e86a2d6287c5" gracePeriod=25 Nov 02 01:35:20.266545 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:20.266512 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e06f2-855bdc6447-dxbs5"] Nov 02 01:35:20.271297 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:20.271278 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e06f2-855bdc6447-dxbs5" Nov 02 01:35:20.280066 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:20.280041 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-0e06f2\"" Nov 02 01:35:20.280066 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:20.280055 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-0e06f2-dockercfg-5vb9d\"" Nov 02 01:35:20.291427 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:20.291395 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e06f2-855bdc6447-dxbs5"] Nov 02 01:35:20.319516 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:20.312961 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-0e06f2-64fc8f586c-8sw9n"] Nov 02 01:35:20.325263 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:20.323829 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e06f2-64fc8f586c-8sw9n" Nov 02 01:35:20.330170 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:20.330146 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-0e06f2-64fc8f586c-8sw9n"] Nov 02 01:35:20.407871 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:20.407825 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79fb94dd-4426-43ed-af58-cfa762e77b13-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-0e06f2-64fc8f586c-8sw9n\" (UID: \"79fb94dd-4426-43ed-af58-cfa762e77b13\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e06f2-64fc8f586c-8sw9n" Nov 02 01:35:20.408065 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:20.407905 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h2q6\" (UniqueName: \"kubernetes.io/projected/aceec71b-ffcb-47e7-964e-84bc8405f2de-kube-api-access-2h2q6\") pod \"payload-pre-processing-e2e-shared-b-w3-0e06f2-855bdc6447-dxbs5\" (UID: \"aceec71b-ffcb-47e7-964e-84bc8405f2de\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e06f2-855bdc6447-dxbs5" Nov 02 01:35:20.408065 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:20.407934 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aceec71b-ffcb-47e7-964e-84bc8405f2de-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-0e06f2-855bdc6447-dxbs5\" (UID: \"aceec71b-ffcb-47e7-964e-84bc8405f2de\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e06f2-855bdc6447-dxbs5" Nov 02 01:35:20.408176 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:20.408080 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbtq7\" (UniqueName: \"kubernetes.io/projected/79fb94dd-4426-43ed-af58-cfa762e77b13-kube-api-access-mbtq7\") pod \"payload-processing-e2e-shared-b-w3-0e06f2-64fc8f586c-8sw9n\" (UID: \"79fb94dd-4426-43ed-af58-cfa762e77b13\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e06f2-64fc8f586c-8sw9n" Nov 02 01:35:20.509859 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:20.509824 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79fb94dd-4426-43ed-af58-cfa762e77b13-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-0e06f2-64fc8f586c-8sw9n\" (UID: \"79fb94dd-4426-43ed-af58-cfa762e77b13\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e06f2-64fc8f586c-8sw9n" Nov 02 01:35:20.510036 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:20.509894 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h2q6\" (UniqueName: \"kubernetes.io/projected/aceec71b-ffcb-47e7-964e-84bc8405f2de-kube-api-access-2h2q6\") pod \"payload-pre-processing-e2e-shared-b-w3-0e06f2-855bdc6447-dxbs5\" (UID: \"aceec71b-ffcb-47e7-964e-84bc8405f2de\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e06f2-855bdc6447-dxbs5" Nov 02 01:35:20.510036 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:20.509929 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aceec71b-ffcb-47e7-964e-84bc8405f2de-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-0e06f2-855bdc6447-dxbs5\" (UID: \"aceec71b-ffcb-47e7-964e-84bc8405f2de\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e06f2-855bdc6447-dxbs5" Nov 02 01:35:20.510149 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:20.510132 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbtq7\" (UniqueName: \"kubernetes.io/projected/79fb94dd-4426-43ed-af58-cfa762e77b13-kube-api-access-mbtq7\") pod \"payload-processing-e2e-shared-b-w3-0e06f2-64fc8f586c-8sw9n\" (UID: \"79fb94dd-4426-43ed-af58-cfa762e77b13\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e06f2-64fc8f586c-8sw9n" Nov 02 01:35:20.510676 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:20.510652 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79fb94dd-4426-43ed-af58-cfa762e77b13-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-0e06f2-64fc8f586c-8sw9n\" (UID: \"79fb94dd-4426-43ed-af58-cfa762e77b13\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e06f2-64fc8f586c-8sw9n" Nov 02 01:35:20.510956 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:20.510933 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aceec71b-ffcb-47e7-964e-84bc8405f2de-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-0e06f2-855bdc6447-dxbs5\" (UID: \"aceec71b-ffcb-47e7-964e-84bc8405f2de\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e06f2-855bdc6447-dxbs5" Nov 02 01:35:20.523164 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:20.523052 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbtq7\" (UniqueName: \"kubernetes.io/projected/79fb94dd-4426-43ed-af58-cfa762e77b13-kube-api-access-mbtq7\") pod \"payload-processing-e2e-shared-b-w3-0e06f2-64fc8f586c-8sw9n\" (UID: \"79fb94dd-4426-43ed-af58-cfa762e77b13\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e06f2-64fc8f586c-8sw9n" Nov 02 01:35:20.523164 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:20.523114 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h2q6\" (UniqueName: \"kubernetes.io/projected/aceec71b-ffcb-47e7-964e-84bc8405f2de-kube-api-access-2h2q6\") pod \"payload-pre-processing-e2e-shared-b-w3-0e06f2-855bdc6447-dxbs5\" (UID: \"aceec71b-ffcb-47e7-964e-84bc8405f2de\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e06f2-855bdc6447-dxbs5" Nov 02 01:35:20.605268 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:20.604968 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e06f2-855bdc6447-dxbs5" Nov 02 01:35:20.650225 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:20.650001 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e06f2-64fc8f586c-8sw9n" Nov 02 01:35:21.122749 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:21.122722 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e06f2-855bdc6447-dxbs5"] Nov 02 01:35:21.122889 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:35:21.122821 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaceec71b_ffcb_47e7_964e_84bc8405f2de.slice/crio-f48343db89be37693f4127e0d60e698b96dbca46122a2f2c91efceb66577ac3d WatchSource:0}: Error finding container f48343db89be37693f4127e0d60e698b96dbca46122a2f2c91efceb66577ac3d: Status 404 returned error can't find the container with id f48343db89be37693f4127e0d60e698b96dbca46122a2f2c91efceb66577ac3d Nov 02 01:35:21.151996 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:21.151968 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-0e06f2-64fc8f586c-8sw9n"] Nov 02 01:35:22.089572 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:22.089535 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e06f2-64fc8f586c-8sw9n" event={"ID":"79fb94dd-4426-43ed-af58-cfa762e77b13","Type":"ContainerStarted","Data":"d7c4ee8f3187db030a01706818be0cad0f93975643eef0c147bd0ba2ba6e98dc"} Nov 02 01:35:22.090065 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:22.089593 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e06f2-64fc8f586c-8sw9n" event={"ID":"79fb94dd-4426-43ed-af58-cfa762e77b13","Type":"ContainerStarted","Data":"2c0c79d5c8b8a7f0778c16d8955211a18e42875f0fdc8c6a894dca9edd8aa9db"} Nov 02 01:35:22.090065 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:22.089637 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e06f2-64fc8f586c-8sw9n" Nov 02 01:35:22.091666 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:22.091637 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e06f2-855bdc6447-dxbs5" event={"ID":"aceec71b-ffcb-47e7-964e-84bc8405f2de","Type":"ContainerStarted","Data":"9f85df429f76202a4141848bceefc109dd8f3ee26d5e6b2333455bf8f40c719f"} Nov 02 01:35:22.091809 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:22.091685 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e06f2-855bdc6447-dxbs5" event={"ID":"aceec71b-ffcb-47e7-964e-84bc8405f2de","Type":"ContainerStarted","Data":"f48343db89be37693f4127e0d60e698b96dbca46122a2f2c91efceb66577ac3d"} Nov 02 01:35:22.091809 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:22.091710 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e06f2-855bdc6447-dxbs5" Nov 02 01:35:22.112412 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:22.112328 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e06f2-64fc8f586c-8sw9n" podStartSLOduration=2.112306575 podStartE2EDuration="2.112306575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:35:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:35:22.109059605 +0000 UTC m=+1226.730064811" watchObservedRunningTime="2026-11-02 01:35:22.112306575 +0000 UTC m=+1226.733311787" Nov 02 01:35:22.138131 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:22.138076 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e06f2-855bdc6447-dxbs5" podStartSLOduration=2.138062152 podStartE2EDuration="2.138062152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:35:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:35:22.134039701 +0000 UTC m=+1226.755044913" watchObservedRunningTime="2026-11-02 01:35:22.138062152 +0000 UTC m=+1226.759067368" Nov 02 01:35:23.794996 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:23.794956 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-50537f4f-67d6cc6b87-kqzbs" podUID="d7877d44-4cab-4c19-a7f1-d761b2be0bb6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 01:35:28.472836 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:28.472799 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-addad9-7579959b99-l926m"] Nov 02 01:35:28.478627 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:28.478606 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-addad9-7579959b99-l926m" Nov 02 01:35:28.484267 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:28.484236 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-addad9\"" Nov 02 01:35:28.484869 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:28.484847 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-addad9-dockercfg-qcdf5\"" Nov 02 01:35:28.487312 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:28.487288 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-addad9-7579959b99-l926m"] Nov 02 01:35:28.570916 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:28.569925 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lwzv\" (UniqueName: \"kubernetes.io/projected/3d9b8d38-b3ce-4556-8580-93cfd36b2d2a-kube-api-access-4lwzv\") pod \"payload-pre-processing-e2e-inf-a-addad9-7579959b99-l926m\" (UID: \"3d9b8d38-b3ce-4556-8580-93cfd36b2d2a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-addad9-7579959b99-l926m" Nov 02 01:35:28.570916 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:28.570225 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d9b8d38-b3ce-4556-8580-93cfd36b2d2a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-addad9-7579959b99-l926m\" (UID: \"3d9b8d38-b3ce-4556-8580-93cfd36b2d2a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-addad9-7579959b99-l926m" Nov 02 01:35:28.710401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:28.707317 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lwzv\" (UniqueName: \"kubernetes.io/projected/3d9b8d38-b3ce-4556-8580-93cfd36b2d2a-kube-api-access-4lwzv\") pod \"payload-pre-processing-e2e-inf-a-addad9-7579959b99-l926m\" (UID: \"3d9b8d38-b3ce-4556-8580-93cfd36b2d2a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-addad9-7579959b99-l926m" Nov 02 01:35:28.710401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:28.707581 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d9b8d38-b3ce-4556-8580-93cfd36b2d2a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-addad9-7579959b99-l926m\" (UID: \"3d9b8d38-b3ce-4556-8580-93cfd36b2d2a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-addad9-7579959b99-l926m" Nov 02 01:35:28.710401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:28.708768 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d9b8d38-b3ce-4556-8580-93cfd36b2d2a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-addad9-7579959b99-l926m\" (UID: \"3d9b8d38-b3ce-4556-8580-93cfd36b2d2a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-addad9-7579959b99-l926m" Nov 02 01:35:28.795267 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:28.795153 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lwzv\" (UniqueName: \"kubernetes.io/projected/3d9b8d38-b3ce-4556-8580-93cfd36b2d2a-kube-api-access-4lwzv\") pod \"payload-pre-processing-e2e-inf-a-addad9-7579959b99-l926m\" (UID: \"3d9b8d38-b3ce-4556-8580-93cfd36b2d2a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-addad9-7579959b99-l926m" Nov 02 01:35:28.806054 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:28.800714 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-addad9-7579959b99-l926m" Nov 02 01:35:29.116610 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:29.116579 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-addad9-7579959b99-l926m"] Nov 02 01:35:29.117799 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:35:29.117764 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d9b8d38_b3ce_4556_8580_93cfd36b2d2a.slice/crio-625b5c066dcf4e45ff905c05ee2a0e01b80647620d658efa5ecfb693e21b7a19 WatchSource:0}: Error finding container 625b5c066dcf4e45ff905c05ee2a0e01b80647620d658efa5ecfb693e21b7a19: Status 404 returned error can't find the container with id 625b5c066dcf4e45ff905c05ee2a0e01b80647620d658efa5ecfb693e21b7a19 Nov 02 01:35:29.165003 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:29.164972 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-addad9-7579959b99-l926m" event={"ID":"3d9b8d38-b3ce-4556-8580-93cfd36b2d2a","Type":"ContainerStarted","Data":"625b5c066dcf4e45ff905c05ee2a0e01b80647620d658efa5ecfb693e21b7a19"} Nov 02 01:35:30.175260 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:30.175224 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-addad9-7579959b99-l926m" event={"ID":"3d9b8d38-b3ce-4556-8580-93cfd36b2d2a","Type":"ContainerStarted","Data":"ef8e82b2a4a793ac0975e87e30e9b105c4d4e8ed64ae72f586943d4999e84141"} Nov 02 01:35:30.175746 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:30.175298 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-addad9-7579959b99-l926m" Nov 02 01:35:30.197389 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:30.197331 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-addad9-7579959b99-l926m" podStartSLOduration=2.197317715 podStartE2EDuration="2.197317715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:35:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:35:30.19284334 +0000 UTC m=+1234.813848580" watchObservedRunningTime="2026-11-02 01:35:30.197317715 +0000 UTC m=+1234.818322925" Nov 02 01:35:31.594521 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:35:31.594476 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:35:33.102418 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:33.102338 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e06f2-855bdc6447-dxbs5" Nov 02 01:35:33.102927 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:33.102594 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e06f2-64fc8f586c-8sw9n" Nov 02 01:35:33.795278 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:33.795237 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-50537f4f-67d6cc6b87-kqzbs" podUID="d7877d44-4cab-4c19-a7f1-d761b2be0bb6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 01:35:33.909636 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:33.909591 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-00f1da-57d6bbd8d9-b8snz"] Nov 02 01:35:33.915606 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:33.915582 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-00f1da-57d6bbd8d9-b8snz" Nov 02 01:35:33.930905 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:33.930881 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-00f1da-57d6bbd8d9-b8snz"] Nov 02 01:35:33.980397 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:33.973662 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-00f1da\"" Nov 02 01:35:34.016403 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:34.012507 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-00f1da-dockercfg-q6rk9\"" Nov 02 01:35:34.049455 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:34.048749 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bf451ee-151b-46e9-91b7-27021d681c0d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-00f1da-57d6bbd8d9-b8snz\" (UID: \"9bf451ee-151b-46e9-91b7-27021d681c0d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-00f1da-57d6bbd8d9-b8snz" Nov 02 01:35:34.049455 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:34.048828 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk5d4\" (UniqueName: \"kubernetes.io/projected/9bf451ee-151b-46e9-91b7-27021d681c0d-kube-api-access-bk5d4\") pod \"payload-processing-e2e-shared-b-w1-00f1da-57d6bbd8d9-b8snz\" (UID: \"9bf451ee-151b-46e9-91b7-27021d681c0d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-00f1da-57d6bbd8d9-b8snz" Nov 02 01:35:34.155598 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:34.155560 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bf451ee-151b-46e9-91b7-27021d681c0d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-00f1da-57d6bbd8d9-b8snz\" (UID: \"9bf451ee-151b-46e9-91b7-27021d681c0d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-00f1da-57d6bbd8d9-b8snz" Nov 02 01:35:34.156228 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:34.156203 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk5d4\" (UniqueName: \"kubernetes.io/projected/9bf451ee-151b-46e9-91b7-27021d681c0d-kube-api-access-bk5d4\") pod \"payload-processing-e2e-shared-b-w1-00f1da-57d6bbd8d9-b8snz\" (UID: \"9bf451ee-151b-46e9-91b7-27021d681c0d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-00f1da-57d6bbd8d9-b8snz" Nov 02 01:35:34.158049 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:34.158011 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bf451ee-151b-46e9-91b7-27021d681c0d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-00f1da-57d6bbd8d9-b8snz\" (UID: \"9bf451ee-151b-46e9-91b7-27021d681c0d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-00f1da-57d6bbd8d9-b8snz" Nov 02 01:35:34.221142 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:34.221095 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk5d4\" (UniqueName: \"kubernetes.io/projected/9bf451ee-151b-46e9-91b7-27021d681c0d-kube-api-access-bk5d4\") pod \"payload-processing-e2e-shared-b-w1-00f1da-57d6bbd8d9-b8snz\" (UID: \"9bf451ee-151b-46e9-91b7-27021d681c0d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-00f1da-57d6bbd8d9-b8snz" Nov 02 01:35:34.237151 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:34.237121 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-00f1da-57d6bbd8d9-b8snz" Nov 02 01:35:34.598635 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:34.594275 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-00f1da-57d6bbd8d9-b8snz"] Nov 02 01:35:35.255121 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:35.254433 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-00f1da-57d6bbd8d9-b8snz" event={"ID":"9bf451ee-151b-46e9-91b7-27021d681c0d","Type":"ContainerStarted","Data":"5c13de5ce0a03794e3177cf56776834f3de68d6ec32266efae880167d4d3ed48"} Nov 02 01:35:35.255121 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:35.254552 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-00f1da-57d6bbd8d9-b8snz" event={"ID":"9bf451ee-151b-46e9-91b7-27021d681c0d","Type":"ContainerStarted","Data":"45ac0a1de1b873b9d6a31b313661c911c432609f18b52487334294201e730115"} Nov 02 01:35:35.255121 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:35.254613 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-00f1da-57d6bbd8d9-b8snz" Nov 02 01:35:35.275395 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:35.275329 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-00f1da-57d6bbd8d9-b8snz" podStartSLOduration=2.275315209 podStartE2EDuration="2.275315209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:35:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:35:35.274434426 +0000 UTC m=+1239.895439636" watchObservedRunningTime="2026-11-02 01:35:35.275315209 +0000 UTC m=+1239.896320419" Nov 02 01:35:36.350417 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:36.349308 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r"] Nov 02 01:35:36.355626 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:36.355597 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r" Nov 02 01:35:36.366954 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:36.361439 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-ffe5c1-openshift-default-dockercfg-f9w6k\"" Nov 02 01:35:36.369889 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:36.369739 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r"] Nov 02 01:35:36.409836 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:36.409797 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c227e450-ed88-4cb4-a5fd-e88f150d14ab-workload-socket\") pod \"e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r\" (UID: \"c227e450-ed88-4cb4-a5fd-e88f150d14ab\") " pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r" Nov 02 01:35:36.410070 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:36.409857 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c227e450-ed88-4cb4-a5fd-e88f150d14ab-istio-token\") pod \"e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r\" (UID: \"c227e450-ed88-4cb4-a5fd-e88f150d14ab\") " pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r" Nov 02 01:35:36.410070 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:36.409957 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c227e450-ed88-4cb4-a5fd-e88f150d14ab-istio-podinfo\") pod \"e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r\" (UID: \"c227e450-ed88-4cb4-a5fd-e88f150d14ab\") " pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r" Nov 02 01:35:36.410070 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:36.410059 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c227e450-ed88-4cb4-a5fd-e88f150d14ab-credential-socket\") pod \"e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r\" (UID: \"c227e450-ed88-4cb4-a5fd-e88f150d14ab\") " pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r" Nov 02 01:35:36.410240 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:36.410162 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c227e450-ed88-4cb4-a5fd-e88f150d14ab-istio-envoy\") pod \"e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r\" (UID: \"c227e450-ed88-4cb4-a5fd-e88f150d14ab\") " pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r" Nov 02 01:35:36.410240 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:36.410192 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c227e450-ed88-4cb4-a5fd-e88f150d14ab-workload-certs\") pod \"e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r\" (UID: \"c227e450-ed88-4cb4-a5fd-e88f150d14ab\") " pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r" Nov 02 01:35:36.410477 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:36.410250 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c227e450-ed88-4cb4-a5fd-e88f150d14ab-istio-data\") pod \"e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r\" (UID: \"c227e450-ed88-4cb4-a5fd-e88f150d14ab\") " pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r" Nov 02 01:35:36.410477 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:36.410352 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c227e450-ed88-4cb4-a5fd-e88f150d14ab-istiod-ca-cert\") pod \"e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r\" (UID: \"c227e450-ed88-4cb4-a5fd-e88f150d14ab\") " pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r" Nov 02 01:35:36.410598 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:36.410511 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-657hs\" (UniqueName: \"kubernetes.io/projected/c227e450-ed88-4cb4-a5fd-e88f150d14ab-kube-api-access-657hs\") pod \"e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r\" (UID: \"c227e450-ed88-4cb4-a5fd-e88f150d14ab\") " pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r" Nov 02 01:35:36.522607 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:36.522555 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c227e450-ed88-4cb4-a5fd-e88f150d14ab-istio-envoy\") pod \"e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r\" (UID: \"c227e450-ed88-4cb4-a5fd-e88f150d14ab\") " pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r" Nov 02 01:35:36.522888 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:36.522865 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c227e450-ed88-4cb4-a5fd-e88f150d14ab-workload-certs\") pod \"e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r\" (UID: \"c227e450-ed88-4cb4-a5fd-e88f150d14ab\") " pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r" Nov 02 01:35:36.523062 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:36.523033 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c227e450-ed88-4cb4-a5fd-e88f150d14ab-istio-data\") pod \"e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r\" (UID: \"c227e450-ed88-4cb4-a5fd-e88f150d14ab\") " pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r" Nov 02 01:35:36.523250 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:36.523234 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c227e450-ed88-4cb4-a5fd-e88f150d14ab-istiod-ca-cert\") pod \"e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r\" (UID: \"c227e450-ed88-4cb4-a5fd-e88f150d14ab\") " pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r" Nov 02 01:35:36.523484 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:36.523461 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-657hs\" (UniqueName: \"kubernetes.io/projected/c227e450-ed88-4cb4-a5fd-e88f150d14ab-kube-api-access-657hs\") pod \"e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r\" (UID: \"c227e450-ed88-4cb4-a5fd-e88f150d14ab\") " pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r" Nov 02 01:35:36.523678 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:36.523662 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c227e450-ed88-4cb4-a5fd-e88f150d14ab-workload-socket\") pod \"e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r\" (UID: \"c227e450-ed88-4cb4-a5fd-e88f150d14ab\") " pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r" Nov 02 01:35:36.523802 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:36.523787 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c227e450-ed88-4cb4-a5fd-e88f150d14ab-istio-token\") pod \"e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r\" (UID: \"c227e450-ed88-4cb4-a5fd-e88f150d14ab\") " pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r" Nov 02 01:35:36.548704 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:36.548664 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c227e450-ed88-4cb4-a5fd-e88f150d14ab-istio-podinfo\") pod \"e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r\" (UID: \"c227e450-ed88-4cb4-a5fd-e88f150d14ab\") " pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r" Nov 02 01:35:36.549064 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:36.549043 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c227e450-ed88-4cb4-a5fd-e88f150d14ab-credential-socket\") pod \"e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r\" (UID: \"c227e450-ed88-4cb4-a5fd-e88f150d14ab\") " pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r" Nov 02 01:35:36.550145 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:36.550116 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c227e450-ed88-4cb4-a5fd-e88f150d14ab-credential-socket\") pod \"e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r\" (UID: \"c227e450-ed88-4cb4-a5fd-e88f150d14ab\") " pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r" Nov 02 01:35:36.550585 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:36.550564 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c227e450-ed88-4cb4-a5fd-e88f150d14ab-istio-data\") pod \"e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r\" (UID: \"c227e450-ed88-4cb4-a5fd-e88f150d14ab\") " pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r" Nov 02 01:35:36.560397 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:36.551664 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c227e450-ed88-4cb4-a5fd-e88f150d14ab-istiod-ca-cert\") pod \"e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r\" (UID: \"c227e450-ed88-4cb4-a5fd-e88f150d14ab\") " pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r" Nov 02 01:35:36.562493 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:36.551815 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c227e450-ed88-4cb4-a5fd-e88f150d14ab-workload-socket\") pod \"e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r\" (UID: \"c227e450-ed88-4cb4-a5fd-e88f150d14ab\") " pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r" Nov 02 01:35:36.562493 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:36.552334 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c227e450-ed88-4cb4-a5fd-e88f150d14ab-workload-certs\") pod \"e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r\" (UID: \"c227e450-ed88-4cb4-a5fd-e88f150d14ab\") " pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r" Nov 02 01:35:36.613338 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:36.613043 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c227e450-ed88-4cb4-a5fd-e88f150d14ab-istio-podinfo\") pod \"e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r\" (UID: \"c227e450-ed88-4cb4-a5fd-e88f150d14ab\") " pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r" Nov 02 01:35:36.616887 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:36.616595 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c227e450-ed88-4cb4-a5fd-e88f150d14ab-istio-token\") pod \"e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r\" (UID: \"c227e450-ed88-4cb4-a5fd-e88f150d14ab\") " pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r" Nov 02 01:35:36.623395 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:36.622988 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c227e450-ed88-4cb4-a5fd-e88f150d14ab-istio-envoy\") pod \"e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r\" (UID: \"c227e450-ed88-4cb4-a5fd-e88f150d14ab\") " pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r" Nov 02 01:35:36.643426 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:36.630749 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-657hs\" (UniqueName: \"kubernetes.io/projected/c227e450-ed88-4cb4-a5fd-e88f150d14ab-kube-api-access-657hs\") pod \"e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r\" (UID: \"c227e450-ed88-4cb4-a5fd-e88f150d14ab\") " pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r" Nov 02 01:35:36.709403 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:36.706750 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r" Nov 02 01:35:37.134896 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:37.134857 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r"] Nov 02 01:35:37.139815 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:37.139773 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:35:37.139955 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:37.139861 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:35:37.139955 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:37.139896 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:35:37.296193 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:37.296160 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r" event={"ID":"c227e450-ed88-4cb4-a5fd-e88f150d14ab","Type":"ContainerStarted","Data":"b572c45e63c2baf75b3cbe98f904258edf95b1dde3445abb8a01ea161ae61022"} Nov 02 01:35:37.296539 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:37.296505 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r" event={"ID":"c227e450-ed88-4cb4-a5fd-e88f150d14ab","Type":"ContainerStarted","Data":"1ae611910ee496162f4a5c4e6c381fd4fb6c926fa36fc442de326517682b31e4"} Nov 02 01:35:37.708158 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:37.708118 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r" Nov 02 01:35:38.498980 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:38.498948 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-50537f4f-67d6cc6b87-kqzbs" Nov 02 01:35:38.520978 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:38.520944 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7877d44-4cab-4c19-a7f1-d761b2be0bb6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7877d44-4cab-4c19-a7f1-d761b2be0bb6-plugins-config-volume\") pod \"d7877d44-4cab-4c19-a7f1-d761b2be0bb6\" (UID: \"d7877d44-4cab-4c19-a7f1-d761b2be0bb6\") " Nov 02 01:35:38.521172 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:38.521133 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7877d44-4cab-4c19-a7f1-d761b2be0bb6-plugins-config-volume" pod "d7877d44-4cab-4c19-a7f1-d761b2be0bb6" (UID: "d7877d44-4cab-4c19-a7f1-d761b2be0bb6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:35:38.521172 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:38.521152 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7877d44-4cab-4c19-a7f1-d761b2be0bb6-kube-api-access-vwkfl\" (UniqueName: \"kubernetes.io/projected/d7877d44-4cab-4c19-a7f1-d761b2be0bb6-kube-api-access-vwkfl\") pod \"d7877d44-4cab-4c19-a7f1-d761b2be0bb6\" (UID: \"d7877d44-4cab-4c19-a7f1-d761b2be0bb6\") " Nov 02 01:35:38.521722 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:38.521701 4427 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7877d44-4cab-4c19-a7f1-d761b2be0bb6-plugins-config-volume\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:35:38.523443 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:38.523417 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7877d44-4cab-4c19-a7f1-d761b2be0bb6-kube-api-access-vwkfl" pod "d7877d44-4cab-4c19-a7f1-d761b2be0bb6" (UID: "d7877d44-4cab-4c19-a7f1-d761b2be0bb6"). InnerVolumeSpecName "kube-api-access-vwkfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:35:38.568559 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:38.563858 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r" podStartSLOduration=2.563835898 podStartE2EDuration="2.563835898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:35:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:35:37.324795365 +0000 UTC m=+1241.945800591" watchObservedRunningTime="2026-11-02 01:35:38.563835898 +0000 UTC m=+1243.184841110" Nov 02 01:35:38.623954 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:38.623921 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwkfl\" (UniqueName: \"kubernetes.io/projected/d7877d44-4cab-4c19-a7f1-d761b2be0bb6-kube-api-access-vwkfl\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:35:38.715324 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:38.715229 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r" Nov 02 01:35:39.343121 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:39.343089 4427 generic.go:356] "Generic (PLEG): container finished" podID="d7877d44-4cab-4c19-a7f1-d761b2be0bb6" containerID="e0c0a7f318355c0d5c099cfdea9e8e9c2f2e237a8ac65493a667e86a2d6287c5" exitCode=0 Nov 02 01:35:39.345152 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:39.344956 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-50537f4f-67d6cc6b87-kqzbs" Nov 02 01:35:39.345712 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:39.345685 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-50537f4f-67d6cc6b87-kqzbs" event={"ID":"d7877d44-4cab-4c19-a7f1-d761b2be0bb6","Type":"ContainerDied","Data":"e0c0a7f318355c0d5c099cfdea9e8e9c2f2e237a8ac65493a667e86a2d6287c5"} Nov 02 01:35:39.345895 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:39.345749 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r" Nov 02 01:35:39.345895 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:39.345777 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-50537f4f-67d6cc6b87-kqzbs" event={"ID":"d7877d44-4cab-4c19-a7f1-d761b2be0bb6","Type":"ContainerDied","Data":"142845e727f9d33122761b783c5bf0ec972db8b935283ccdeb3d39d975704cd6"} Nov 02 01:35:39.345895 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:39.345806 4427 scope.go:122] "RemoveContainer" containerID="e0c0a7f318355c0d5c099cfdea9e8e9c2f2e237a8ac65493a667e86a2d6287c5" Nov 02 01:35:39.346984 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:39.346962 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r" Nov 02 01:35:39.363594 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:39.363576 4427 scope.go:122] "RemoveContainer" containerID="e0c0a7f318355c0d5c099cfdea9e8e9c2f2e237a8ac65493a667e86a2d6287c5" Nov 02 01:35:39.363905 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:35:39.363885 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0c0a7f318355c0d5c099cfdea9e8e9c2f2e237a8ac65493a667e86a2d6287c5\": container with ID starting with e0c0a7f318355c0d5c099cfdea9e8e9c2f2e237a8ac65493a667e86a2d6287c5 not found: ID does not exist" containerID="e0c0a7f318355c0d5c099cfdea9e8e9c2f2e237a8ac65493a667e86a2d6287c5" Nov 02 01:35:39.363973 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:39.363914 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0c0a7f318355c0d5c099cfdea9e8e9c2f2e237a8ac65493a667e86a2d6287c5"} err="failed to get container status \"e0c0a7f318355c0d5c099cfdea9e8e9c2f2e237a8ac65493a667e86a2d6287c5\": rpc error: code = NotFound desc = could not find container \"e0c0a7f318355c0d5c099cfdea9e8e9c2f2e237a8ac65493a667e86a2d6287c5\": container with ID starting with e0c0a7f318355c0d5c099cfdea9e8e9c2f2e237a8ac65493a667e86a2d6287c5 not found: ID does not exist" Nov 02 01:35:39.415446 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:39.407783 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-50537f4f-67d6cc6b87-kqzbs"] Nov 02 01:35:39.415446 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:39.414784 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-50537f4f-67d6cc6b87-kqzbs"] Nov 02 01:35:39.536524 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:39.536491 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7877d44-4cab-4c19-a7f1-d761b2be0bb6" path="/var/lib/kubelet/pods/d7877d44-4cab-4c19-a7f1-d761b2be0bb6/volumes" Nov 02 01:35:39.731326 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:39.731285 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm"] Nov 02 01:35:39.731757 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:39.731529 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm" podUID="104f9b56-ef05-4c25-a485-4bbb6bac1ed0" containerName="istio-proxy" containerID="cri-o://421335e5adc15f5080e7e69849a5454438b4f77a765620f94dc1b0b5e0a3fa6a" gracePeriod=30 Nov 02 01:35:39.736919 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:39.736869 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb"] Nov 02 01:35:39.737196 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:39.737170 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb" podUID="917e0c7d-4eae-4d2d-bb59-f2de47fbfadb" containerName="istio-proxy" containerID="cri-o://ae0e647af21cce922369cf85ae035291361db5082678af5fe817f4f890c8389b" gracePeriod=30 Nov 02 01:35:41.047876 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:41.047832 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ffe5c1-579789c787-hz68t"] Nov 02 01:35:41.049038 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:41.049010 4427 cpu_manager.go:414] "containerMap: removing container" podUID="d7877d44-4cab-4c19-a7f1-d761b2be0bb6" containerName="payload-processing" Nov 02 01:35:41.049038 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:41.049040 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7877d44-4cab-4c19-a7f1-d761b2be0bb6" containerName="payload-processing" Nov 02 01:35:41.049322 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:41.049304 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7877d44-4cab-4c19-a7f1-d761b2be0bb6" containerName="payload-processing" Nov 02 01:35:41.056648 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:41.056623 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ffe5c1-579789c787-hz68t" Nov 02 01:35:41.110150 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:41.108017 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ffe5c1-579789c787-hz68t"] Nov 02 01:35:41.110150 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:41.108423 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-ffe5c1\"" Nov 02 01:35:41.110150 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:41.108721 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-ffe5c1-dockercfg-xn2ws\"" Nov 02 01:35:41.112312 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:41.110970 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-572r8\" (UniqueName: \"kubernetes.io/projected/b9bb7aa5-03de-42b9-a6d3-b1d8e7806a74-kube-api-access-572r8\") pod \"payload-processing-e2e-inf-b-ffe5c1-579789c787-hz68t\" (UID: \"b9bb7aa5-03de-42b9-a6d3-b1d8e7806a74\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ffe5c1-579789c787-hz68t" Nov 02 01:35:41.112312 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:41.111119 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9bb7aa5-03de-42b9-a6d3-b1d8e7806a74-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ffe5c1-579789c787-hz68t\" (UID: \"b9bb7aa5-03de-42b9-a6d3-b1d8e7806a74\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ffe5c1-579789c787-hz68t" Nov 02 01:35:41.234398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:41.229209 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-572r8\" (UniqueName: \"kubernetes.io/projected/b9bb7aa5-03de-42b9-a6d3-b1d8e7806a74-kube-api-access-572r8\") pod \"payload-processing-e2e-inf-b-ffe5c1-579789c787-hz68t\" (UID: \"b9bb7aa5-03de-42b9-a6d3-b1d8e7806a74\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ffe5c1-579789c787-hz68t" Nov 02 01:35:41.234398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:41.229393 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9bb7aa5-03de-42b9-a6d3-b1d8e7806a74-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ffe5c1-579789c787-hz68t\" (UID: \"b9bb7aa5-03de-42b9-a6d3-b1d8e7806a74\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ffe5c1-579789c787-hz68t" Nov 02 01:35:41.234398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:41.230414 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9bb7aa5-03de-42b9-a6d3-b1d8e7806a74-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ffe5c1-579789c787-hz68t\" (UID: \"b9bb7aa5-03de-42b9-a6d3-b1d8e7806a74\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ffe5c1-579789c787-hz68t" Nov 02 01:35:41.252649 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:41.249104 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-addad9-7579959b99-l926m" Nov 02 01:35:41.616508 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:41.614179 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-572r8\" (UniqueName: \"kubernetes.io/projected/b9bb7aa5-03de-42b9-a6d3-b1d8e7806a74-kube-api-access-572r8\") pod \"payload-processing-e2e-inf-b-ffe5c1-579789c787-hz68t\" (UID: \"b9bb7aa5-03de-42b9-a6d3-b1d8e7806a74\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ffe5c1-579789c787-hz68t" Nov 02 01:35:41.734636 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:41.734601 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ffe5c1-579789c787-hz68t" Nov 02 01:35:42.053452 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:42.048519 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ffe5c1-579789c787-hz68t"] Nov 02 01:35:42.058247 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:35:42.058215 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9bb7aa5_03de_42b9_a6d3_b1d8e7806a74.slice/crio-339373e483955ef88e515446956fe0ba62bb5cdf18349e2a6f98b7da721c0d47 WatchSource:0}: Error finding container 339373e483955ef88e515446956fe0ba62bb5cdf18349e2a6f98b7da721c0d47: Status 404 returned error can't find the container with id 339373e483955ef88e515446956fe0ba62bb5cdf18349e2a6f98b7da721c0d47 Nov 02 01:35:42.655643 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:42.655590 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ffe5c1-579789c787-hz68t" event={"ID":"b9bb7aa5-03de-42b9-a6d3-b1d8e7806a74","Type":"ContainerStarted","Data":"de6f1c505982666c93a25403b9eea087c169cd8dac0531c888ef4a3351663a60"} Nov 02 01:35:42.655834 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:42.655665 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ffe5c1-579789c787-hz68t" event={"ID":"b9bb7aa5-03de-42b9-a6d3-b1d8e7806a74","Type":"ContainerStarted","Data":"339373e483955ef88e515446956fe0ba62bb5cdf18349e2a6f98b7da721c0d47"} Nov 02 01:35:42.655834 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:42.655695 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-ffe5c1-579789c787-hz68t" Nov 02 01:35:42.675871 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:42.675816 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-ffe5c1-579789c787-hz68t" podStartSLOduration=1.675801726 podStartE2EDuration="1.675801726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:35:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:35:42.673443332 +0000 UTC m=+1247.294448544" watchObservedRunningTime="2026-11-02 01:35:42.675801726 +0000 UTC m=+1247.296806937" Nov 02 01:35:45.128555 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.128519 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm" Nov 02 01:35:45.132131 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.132110 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb" Nov 02 01:35:45.220650 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.220599 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/104f9b56-ef05-4c25-a485-4bbb6bac1ed0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/104f9b56-ef05-4c25-a485-4bbb6bac1ed0-workload-socket\") pod \"104f9b56-ef05-4c25-a485-4bbb6bac1ed0\" (UID: \"104f9b56-ef05-4c25-a485-4bbb6bac1ed0\") " Nov 02 01:35:45.220844 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.220677 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/917e0c7d-4eae-4d2d-bb59-f2de47fbfadb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/917e0c7d-4eae-4d2d-bb59-f2de47fbfadb-credential-socket\") pod \"917e0c7d-4eae-4d2d-bb59-f2de47fbfadb\" (UID: \"917e0c7d-4eae-4d2d-bb59-f2de47fbfadb\") " Nov 02 01:35:45.220844 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.220736 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/917e0c7d-4eae-4d2d-bb59-f2de47fbfadb-kube-api-access-nzpc2\" (UniqueName: \"kubernetes.io/projected/917e0c7d-4eae-4d2d-bb59-f2de47fbfadb-kube-api-access-nzpc2\") pod \"917e0c7d-4eae-4d2d-bb59-f2de47fbfadb\" (UID: \"917e0c7d-4eae-4d2d-bb59-f2de47fbfadb\") " Nov 02 01:35:45.221019 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.220995 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/104f9b56-ef05-4c25-a485-4bbb6bac1ed0-workload-socket" pod "104f9b56-ef05-4c25-a485-4bbb6bac1ed0" (UID: "104f9b56-ef05-4c25-a485-4bbb6bac1ed0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:35:45.221077 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.221013 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/917e0c7d-4eae-4d2d-bb59-f2de47fbfadb-credential-socket" pod "917e0c7d-4eae-4d2d-bb59-f2de47fbfadb" (UID: "917e0c7d-4eae-4d2d-bb59-f2de47fbfadb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:35:45.221290 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.221270 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/104f9b56-ef05-4c25-a485-4bbb6bac1ed0-workload-certs" pod "104f9b56-ef05-4c25-a485-4bbb6bac1ed0" (UID: "104f9b56-ef05-4c25-a485-4bbb6bac1ed0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:35:45.221436 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.221273 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/104f9b56-ef05-4c25-a485-4bbb6bac1ed0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/104f9b56-ef05-4c25-a485-4bbb6bac1ed0-workload-certs\") pod \"104f9b56-ef05-4c25-a485-4bbb6bac1ed0\" (UID: \"104f9b56-ef05-4c25-a485-4bbb6bac1ed0\") " Nov 02 01:35:45.221436 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.221344 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/104f9b56-ef05-4c25-a485-4bbb6bac1ed0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/104f9b56-ef05-4c25-a485-4bbb6bac1ed0-istio-data\") pod \"104f9b56-ef05-4c25-a485-4bbb6bac1ed0\" (UID: \"104f9b56-ef05-4c25-a485-4bbb6bac1ed0\") " Nov 02 01:35:45.221584 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.221451 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/104f9b56-ef05-4c25-a485-4bbb6bac1ed0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/104f9b56-ef05-4c25-a485-4bbb6bac1ed0-istio-envoy\") pod \"104f9b56-ef05-4c25-a485-4bbb6bac1ed0\" (UID: \"104f9b56-ef05-4c25-a485-4bbb6bac1ed0\") " Nov 02 01:35:45.221650 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.221627 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/104f9b56-ef05-4c25-a485-4bbb6bac1ed0-istio-data" pod "104f9b56-ef05-4c25-a485-4bbb6bac1ed0" (UID: "104f9b56-ef05-4c25-a485-4bbb6bac1ed0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:35:45.221846 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.221826 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/917e0c7d-4eae-4d2d-bb59-f2de47fbfadb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/917e0c7d-4eae-4d2d-bb59-f2de47fbfadb-workload-certs\") pod \"917e0c7d-4eae-4d2d-bb59-f2de47fbfadb\" (UID: \"917e0c7d-4eae-4d2d-bb59-f2de47fbfadb\") " Nov 02 01:35:45.221922 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.221910 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/917e0c7d-4eae-4d2d-bb59-f2de47fbfadb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/917e0c7d-4eae-4d2d-bb59-f2de47fbfadb-istio-podinfo\") pod \"917e0c7d-4eae-4d2d-bb59-f2de47fbfadb\" (UID: \"917e0c7d-4eae-4d2d-bb59-f2de47fbfadb\") " Nov 02 01:35:45.222013 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.221995 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/104f9b56-ef05-4c25-a485-4bbb6bac1ed0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/104f9b56-ef05-4c25-a485-4bbb6bac1ed0-istio-podinfo\") pod \"104f9b56-ef05-4c25-a485-4bbb6bac1ed0\" (UID: \"104f9b56-ef05-4c25-a485-4bbb6bac1ed0\") " Nov 02 01:35:45.222093 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.222078 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/104f9b56-ef05-4c25-a485-4bbb6bac1ed0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/104f9b56-ef05-4c25-a485-4bbb6bac1ed0-istiod-ca-cert\") pod \"104f9b56-ef05-4c25-a485-4bbb6bac1ed0\" (UID: \"104f9b56-ef05-4c25-a485-4bbb6bac1ed0\") " Nov 02 01:35:45.222144 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.222085 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/917e0c7d-4eae-4d2d-bb59-f2de47fbfadb-workload-certs" pod "917e0c7d-4eae-4d2d-bb59-f2de47fbfadb" (UID: "917e0c7d-4eae-4d2d-bb59-f2de47fbfadb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:35:45.222247 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.222228 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/917e0c7d-4eae-4d2d-bb59-f2de47fbfadb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/917e0c7d-4eae-4d2d-bb59-f2de47fbfadb-istio-data\") pod \"917e0c7d-4eae-4d2d-bb59-f2de47fbfadb\" (UID: \"917e0c7d-4eae-4d2d-bb59-f2de47fbfadb\") " Nov 02 01:35:45.222514 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.222280 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/917e0c7d-4eae-4d2d-bb59-f2de47fbfadb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/917e0c7d-4eae-4d2d-bb59-f2de47fbfadb-istiod-ca-cert\") pod \"917e0c7d-4eae-4d2d-bb59-f2de47fbfadb\" (UID: \"917e0c7d-4eae-4d2d-bb59-f2de47fbfadb\") " Nov 02 01:35:45.222514 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.222345 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/104f9b56-ef05-4c25-a485-4bbb6bac1ed0-kube-api-access-kq7bp\" (UniqueName: \"kubernetes.io/projected/104f9b56-ef05-4c25-a485-4bbb6bac1ed0-kube-api-access-kq7bp\") pod \"104f9b56-ef05-4c25-a485-4bbb6bac1ed0\" (UID: \"104f9b56-ef05-4c25-a485-4bbb6bac1ed0\") " Nov 02 01:35:45.222514 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.222420 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/104f9b56-ef05-4c25-a485-4bbb6bac1ed0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/104f9b56-ef05-4c25-a485-4bbb6bac1ed0-credential-socket\") pod \"104f9b56-ef05-4c25-a485-4bbb6bac1ed0\" (UID: \"104f9b56-ef05-4c25-a485-4bbb6bac1ed0\") " Nov 02 01:35:45.222773 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.222747 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/104f9b56-ef05-4c25-a485-4bbb6bac1ed0-istiod-ca-cert" pod "104f9b56-ef05-4c25-a485-4bbb6bac1ed0" (UID: "104f9b56-ef05-4c25-a485-4bbb6bac1ed0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:35:45.223337 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.223312 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/917e0c7d-4eae-4d2d-bb59-f2de47fbfadb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/917e0c7d-4eae-4d2d-bb59-f2de47fbfadb-istio-envoy\") pod \"917e0c7d-4eae-4d2d-bb59-f2de47fbfadb\" (UID: \"917e0c7d-4eae-4d2d-bb59-f2de47fbfadb\") " Nov 02 01:35:45.223450 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.223392 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/917e0c7d-4eae-4d2d-bb59-f2de47fbfadb-istiod-ca-cert" pod "917e0c7d-4eae-4d2d-bb59-f2de47fbfadb" (UID: "917e0c7d-4eae-4d2d-bb59-f2de47fbfadb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:35:45.223450 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.223418 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/917e0c7d-4eae-4d2d-bb59-f2de47fbfadb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/917e0c7d-4eae-4d2d-bb59-f2de47fbfadb-workload-socket\") pod \"917e0c7d-4eae-4d2d-bb59-f2de47fbfadb\" (UID: \"917e0c7d-4eae-4d2d-bb59-f2de47fbfadb\") " Nov 02 01:35:45.223559 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.223469 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/917e0c7d-4eae-4d2d-bb59-f2de47fbfadb-istio-token\" (UniqueName: \"kubernetes.io/projected/917e0c7d-4eae-4d2d-bb59-f2de47fbfadb-istio-token\") pod \"917e0c7d-4eae-4d2d-bb59-f2de47fbfadb\" (UID: \"917e0c7d-4eae-4d2d-bb59-f2de47fbfadb\") " Nov 02 01:35:45.223559 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.223505 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/104f9b56-ef05-4c25-a485-4bbb6bac1ed0-istio-token\" (UniqueName: \"kubernetes.io/projected/104f9b56-ef05-4c25-a485-4bbb6bac1ed0-istio-token\") pod \"104f9b56-ef05-4c25-a485-4bbb6bac1ed0\" (UID: \"104f9b56-ef05-4c25-a485-4bbb6bac1ed0\") " Nov 02 01:35:45.223650 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.223620 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/917e0c7d-4eae-4d2d-bb59-f2de47fbfadb-istio-data" pod "917e0c7d-4eae-4d2d-bb59-f2de47fbfadb" (UID: "917e0c7d-4eae-4d2d-bb59-f2de47fbfadb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:35:45.224528 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.224498 4427 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/917e0c7d-4eae-4d2d-bb59-f2de47fbfadb-credential-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:35:45.224610 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.224529 4427 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/104f9b56-ef05-4c25-a485-4bbb6bac1ed0-workload-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:35:45.224610 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.224544 4427 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/104f9b56-ef05-4c25-a485-4bbb6bac1ed0-istio-data\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:35:45.224610 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.224557 4427 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/917e0c7d-4eae-4d2d-bb59-f2de47fbfadb-workload-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:35:45.224610 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.224570 4427 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/104f9b56-ef05-4c25-a485-4bbb6bac1ed0-istiod-ca-cert\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:35:45.224610 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.224582 4427 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/917e0c7d-4eae-4d2d-bb59-f2de47fbfadb-istio-data\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:35:45.224610 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.224595 4427 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/917e0c7d-4eae-4d2d-bb59-f2de47fbfadb-istiod-ca-cert\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:35:45.224610 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.224609 4427 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/104f9b56-ef05-4c25-a485-4bbb6bac1ed0-workload-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:35:45.224862 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.224833 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/917e0c7d-4eae-4d2d-bb59-f2de47fbfadb-workload-socket" pod "917e0c7d-4eae-4d2d-bb59-f2de47fbfadb" (UID: "917e0c7d-4eae-4d2d-bb59-f2de47fbfadb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:35:45.226011 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.225981 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/104f9b56-ef05-4c25-a485-4bbb6bac1ed0-istio-envoy" pod "104f9b56-ef05-4c25-a485-4bbb6bac1ed0" (UID: "104f9b56-ef05-4c25-a485-4bbb6bac1ed0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:35:45.226286 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.226263 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/104f9b56-ef05-4c25-a485-4bbb6bac1ed0-credential-socket" pod "104f9b56-ef05-4c25-a485-4bbb6bac1ed0" (UID: "104f9b56-ef05-4c25-a485-4bbb6bac1ed0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:35:45.226699 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.226672 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/917e0c7d-4eae-4d2d-bb59-f2de47fbfadb-kube-api-access-nzpc2" pod "917e0c7d-4eae-4d2d-bb59-f2de47fbfadb" (UID: "917e0c7d-4eae-4d2d-bb59-f2de47fbfadb"). InnerVolumeSpecName "kube-api-access-nzpc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:35:45.228844 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.228820 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/917e0c7d-4eae-4d2d-bb59-f2de47fbfadb-istio-token" pod "917e0c7d-4eae-4d2d-bb59-f2de47fbfadb" (UID: "917e0c7d-4eae-4d2d-bb59-f2de47fbfadb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:35:45.229767 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.229737 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/917e0c7d-4eae-4d2d-bb59-f2de47fbfadb-istio-envoy" pod "917e0c7d-4eae-4d2d-bb59-f2de47fbfadb" (UID: "917e0c7d-4eae-4d2d-bb59-f2de47fbfadb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:35:45.229767 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.229740 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/104f9b56-ef05-4c25-a485-4bbb6bac1ed0-istio-podinfo" pod "104f9b56-ef05-4c25-a485-4bbb6bac1ed0" (UID: "104f9b56-ef05-4c25-a485-4bbb6bac1ed0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:35:45.229892 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.229763 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/917e0c7d-4eae-4d2d-bb59-f2de47fbfadb-istio-podinfo" pod "917e0c7d-4eae-4d2d-bb59-f2de47fbfadb" (UID: "917e0c7d-4eae-4d2d-bb59-f2de47fbfadb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:35:45.229944 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.229920 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/104f9b56-ef05-4c25-a485-4bbb6bac1ed0-kube-api-access-kq7bp" pod "104f9b56-ef05-4c25-a485-4bbb6bac1ed0" (UID: "104f9b56-ef05-4c25-a485-4bbb6bac1ed0"). InnerVolumeSpecName "kube-api-access-kq7bp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:35:45.231069 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.231046 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/104f9b56-ef05-4c25-a485-4bbb6bac1ed0-istio-token" pod "104f9b56-ef05-4c25-a485-4bbb6bac1ed0" (UID: "104f9b56-ef05-4c25-a485-4bbb6bac1ed0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:35:45.326277 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.326240 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kq7bp\" (UniqueName: \"kubernetes.io/projected/104f9b56-ef05-4c25-a485-4bbb6bac1ed0-kube-api-access-kq7bp\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:35:45.326277 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.326279 4427 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/104f9b56-ef05-4c25-a485-4bbb6bac1ed0-credential-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:35:45.326571 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.326292 4427 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/917e0c7d-4eae-4d2d-bb59-f2de47fbfadb-istio-envoy\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:35:45.326571 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.326306 4427 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/917e0c7d-4eae-4d2d-bb59-f2de47fbfadb-workload-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:35:45.326571 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.326318 4427 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/917e0c7d-4eae-4d2d-bb59-f2de47fbfadb-istio-token\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:35:45.326571 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.326332 4427 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/104f9b56-ef05-4c25-a485-4bbb6bac1ed0-istio-token\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:35:45.326571 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.326344 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzpc2\" (UniqueName: \"kubernetes.io/projected/917e0c7d-4eae-4d2d-bb59-f2de47fbfadb-kube-api-access-nzpc2\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:35:45.326571 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.326355 4427 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/104f9b56-ef05-4c25-a485-4bbb6bac1ed0-istio-envoy\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:35:45.326571 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.326367 4427 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/917e0c7d-4eae-4d2d-bb59-f2de47fbfadb-istio-podinfo\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:35:45.326571 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.326428 4427 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/104f9b56-ef05-4c25-a485-4bbb6bac1ed0-istio-podinfo\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:35:45.687321 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.687217 4427 generic.go:356] "Generic (PLEG): container finished" podID="917e0c7d-4eae-4d2d-bb59-f2de47fbfadb" containerID="ae0e647af21cce922369cf85ae035291361db5082678af5fe817f4f890c8389b" exitCode=0 Nov 02 01:35:45.687321 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.687270 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb" Nov 02 01:35:45.687629 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.687507 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb" event={"ID":"917e0c7d-4eae-4d2d-bb59-f2de47fbfadb","Type":"ContainerDied","Data":"ae0e647af21cce922369cf85ae035291361db5082678af5fe817f4f890c8389b"} Nov 02 01:35:45.687629 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.687585 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb" event={"ID":"917e0c7d-4eae-4d2d-bb59-f2de47fbfadb","Type":"ContainerDied","Data":"6d58a54b89213f35dbac1aeead851b72347cf109337710d125e6f4b7056af954"} Nov 02 01:35:45.687629 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.687623 4427 scope.go:122] "RemoveContainer" containerID="ae0e647af21cce922369cf85ae035291361db5082678af5fe817f4f890c8389b" Nov 02 01:35:45.691427 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.691344 4427 generic.go:356] "Generic (PLEG): container finished" podID="104f9b56-ef05-4c25-a485-4bbb6bac1ed0" containerID="421335e5adc15f5080e7e69849a5454438b4f77a765620f94dc1b0b5e0a3fa6a" exitCode=0 Nov 02 01:35:45.691568 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.691423 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm" event={"ID":"104f9b56-ef05-4c25-a485-4bbb6bac1ed0","Type":"ContainerDied","Data":"421335e5adc15f5080e7e69849a5454438b4f77a765620f94dc1b0b5e0a3fa6a"} Nov 02 01:35:45.691568 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.691483 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm" event={"ID":"104f9b56-ef05-4c25-a485-4bbb6bac1ed0","Type":"ContainerDied","Data":"2008039fd6c3bfcdae4c5d052caf004a6a3a093c3ba7dc4008cc83105209f90e"} Nov 02 01:35:45.691568 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.691532 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm" Nov 02 01:35:45.700728 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.700709 4427 scope.go:122] "RemoveContainer" containerID="ae0e647af21cce922369cf85ae035291361db5082678af5fe817f4f890c8389b" Nov 02 01:35:45.701053 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:35:45.701027 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae0e647af21cce922369cf85ae035291361db5082678af5fe817f4f890c8389b\": container with ID starting with ae0e647af21cce922369cf85ae035291361db5082678af5fe817f4f890c8389b not found: ID does not exist" containerID="ae0e647af21cce922369cf85ae035291361db5082678af5fe817f4f890c8389b" Nov 02 01:35:45.701119 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.701066 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae0e647af21cce922369cf85ae035291361db5082678af5fe817f4f890c8389b"} err="failed to get container status \"ae0e647af21cce922369cf85ae035291361db5082678af5fe817f4f890c8389b\": rpc error: code = NotFound desc = could not find container \"ae0e647af21cce922369cf85ae035291361db5082678af5fe817f4f890c8389b\": container with ID starting with ae0e647af21cce922369cf85ae035291361db5082678af5fe817f4f890c8389b not found: ID does not exist" Nov 02 01:35:45.701119 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.701090 4427 scope.go:122] "RemoveContainer" containerID="421335e5adc15f5080e7e69849a5454438b4f77a765620f94dc1b0b5e0a3fa6a" Nov 02 01:35:45.714450 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.714362 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb"] Nov 02 01:35:45.719761 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.719731 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-qbmdb"] Nov 02 01:35:45.720215 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.720195 4427 scope.go:122] "RemoveContainer" containerID="421335e5adc15f5080e7e69849a5454438b4f77a765620f94dc1b0b5e0a3fa6a" Nov 02 01:35:45.720580 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:35:45.720560 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"421335e5adc15f5080e7e69849a5454438b4f77a765620f94dc1b0b5e0a3fa6a\": container with ID starting with 421335e5adc15f5080e7e69849a5454438b4f77a765620f94dc1b0b5e0a3fa6a not found: ID does not exist" containerID="421335e5adc15f5080e7e69849a5454438b4f77a765620f94dc1b0b5e0a3fa6a" Nov 02 01:35:45.720650 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.720588 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"421335e5adc15f5080e7e69849a5454438b4f77a765620f94dc1b0b5e0a3fa6a"} err="failed to get container status \"421335e5adc15f5080e7e69849a5454438b4f77a765620f94dc1b0b5e0a3fa6a\": rpc error: code = NotFound desc = could not find container \"421335e5adc15f5080e7e69849a5454438b4f77a765620f94dc1b0b5e0a3fa6a\": container with ID starting with 421335e5adc15f5080e7e69849a5454438b4f77a765620f94dc1b0b5e0a3fa6a not found: ID does not exist" Nov 02 01:35:45.734682 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.734659 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm"] Nov 02 01:35:45.738698 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:45.738677 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-ftdkm"] Nov 02 01:35:46.268214 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:46.268184 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-00f1da-57d6bbd8d9-b8snz" Nov 02 01:35:47.539019 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:47.538975 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="104f9b56-ef05-4c25-a485-4bbb6bac1ed0" path="/var/lib/kubelet/pods/104f9b56-ef05-4c25-a485-4bbb6bac1ed0/volumes" Nov 02 01:35:47.539788 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:47.539764 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="917e0c7d-4eae-4d2d-bb59-f2de47fbfadb" path="/var/lib/kubelet/pods/917e0c7d-4eae-4d2d-bb59-f2de47fbfadb/volumes" Nov 02 01:35:50.563398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:50.560977 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4"] Nov 02 01:35:50.563398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:50.561972 4427 cpu_manager.go:414] "containerMap: removing container" podUID="917e0c7d-4eae-4d2d-bb59-f2de47fbfadb" containerName="istio-proxy" Nov 02 01:35:50.563398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:50.561993 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="917e0c7d-4eae-4d2d-bb59-f2de47fbfadb" containerName="istio-proxy" Nov 02 01:35:50.563398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:50.562013 4427 cpu_manager.go:414] "containerMap: removing container" podUID="104f9b56-ef05-4c25-a485-4bbb6bac1ed0" containerName="istio-proxy" Nov 02 01:35:50.563398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:50.562020 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="104f9b56-ef05-4c25-a485-4bbb6bac1ed0" containerName="istio-proxy" Nov 02 01:35:50.563398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:50.562223 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="917e0c7d-4eae-4d2d-bb59-f2de47fbfadb" containerName="istio-proxy" Nov 02 01:35:50.563398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:50.562240 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="104f9b56-ef05-4c25-a485-4bbb6bac1ed0" containerName="istio-proxy" Nov 02 01:35:50.567109 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:50.567083 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4" Nov 02 01:35:50.574138 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:50.574111 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-b34fc0-openshift-default-dockercfg-p7dfb\"" Nov 02 01:35:50.598442 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:50.598121 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4"] Nov 02 01:35:50.653459 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:50.652068 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3d179f1-c7e0-4340-a286-2efbbc44ed53-credential-socket\") pod \"e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4\" (UID: \"c3d179f1-c7e0-4340-a286-2efbbc44ed53\") " pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4" Nov 02 01:35:50.653459 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:50.652119 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3d179f1-c7e0-4340-a286-2efbbc44ed53-workload-certs\") pod \"e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4\" (UID: \"c3d179f1-c7e0-4340-a286-2efbbc44ed53\") " pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4" Nov 02 01:35:50.653459 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:50.652259 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3d179f1-c7e0-4340-a286-2efbbc44ed53-istio-podinfo\") pod \"e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4\" (UID: \"c3d179f1-c7e0-4340-a286-2efbbc44ed53\") " pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4" Nov 02 01:35:50.653459 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:50.652352 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3d179f1-c7e0-4340-a286-2efbbc44ed53-istio-data\") pod \"e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4\" (UID: \"c3d179f1-c7e0-4340-a286-2efbbc44ed53\") " pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4" Nov 02 01:35:50.653459 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:50.652516 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl9pj\" (UniqueName: \"kubernetes.io/projected/c3d179f1-c7e0-4340-a286-2efbbc44ed53-kube-api-access-sl9pj\") pod \"e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4\" (UID: \"c3d179f1-c7e0-4340-a286-2efbbc44ed53\") " pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4" Nov 02 01:35:50.653459 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:50.652569 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3d179f1-c7e0-4340-a286-2efbbc44ed53-istiod-ca-cert\") pod \"e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4\" (UID: \"c3d179f1-c7e0-4340-a286-2efbbc44ed53\") " pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4" Nov 02 01:35:50.653459 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:50.652696 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3d179f1-c7e0-4340-a286-2efbbc44ed53-workload-socket\") pod \"e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4\" (UID: \"c3d179f1-c7e0-4340-a286-2efbbc44ed53\") " pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4" Nov 02 01:35:50.653459 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:50.652884 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3d179f1-c7e0-4340-a286-2efbbc44ed53-istio-envoy\") pod \"e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4\" (UID: \"c3d179f1-c7e0-4340-a286-2efbbc44ed53\") " pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4" Nov 02 01:35:50.653459 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:50.653012 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3d179f1-c7e0-4340-a286-2efbbc44ed53-istio-token\") pod \"e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4\" (UID: \"c3d179f1-c7e0-4340-a286-2efbbc44ed53\") " pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4" Nov 02 01:35:50.786420 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:50.768675 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3d179f1-c7e0-4340-a286-2efbbc44ed53-istiod-ca-cert\") pod \"e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4\" (UID: \"c3d179f1-c7e0-4340-a286-2efbbc44ed53\") " pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4" Nov 02 01:35:50.786420 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:50.768840 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3d179f1-c7e0-4340-a286-2efbbc44ed53-workload-socket\") pod \"e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4\" (UID: \"c3d179f1-c7e0-4340-a286-2efbbc44ed53\") " pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4" Nov 02 01:35:50.786420 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:50.768972 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3d179f1-c7e0-4340-a286-2efbbc44ed53-istio-envoy\") pod \"e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4\" (UID: \"c3d179f1-c7e0-4340-a286-2efbbc44ed53\") " pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4" Nov 02 01:35:50.829465 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:50.822952 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3d179f1-c7e0-4340-a286-2efbbc44ed53-workload-socket\") pod \"e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4\" (UID: \"c3d179f1-c7e0-4340-a286-2efbbc44ed53\") " pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4" Nov 02 01:35:50.829465 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:50.822981 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3d179f1-c7e0-4340-a286-2efbbc44ed53-istiod-ca-cert\") pod \"e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4\" (UID: \"c3d179f1-c7e0-4340-a286-2efbbc44ed53\") " pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4" Nov 02 01:35:50.829465 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:50.823647 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3d179f1-c7e0-4340-a286-2efbbc44ed53-istio-token\") pod \"e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4\" (UID: \"c3d179f1-c7e0-4340-a286-2efbbc44ed53\") " pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4" Nov 02 01:35:50.829465 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:50.823752 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3d179f1-c7e0-4340-a286-2efbbc44ed53-credential-socket\") pod \"e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4\" (UID: \"c3d179f1-c7e0-4340-a286-2efbbc44ed53\") " pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4" Nov 02 01:35:50.829465 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:50.823782 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3d179f1-c7e0-4340-a286-2efbbc44ed53-workload-certs\") pod \"e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4\" (UID: \"c3d179f1-c7e0-4340-a286-2efbbc44ed53\") " pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4" Nov 02 01:35:50.829465 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:50.823856 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3d179f1-c7e0-4340-a286-2efbbc44ed53-istio-podinfo\") pod \"e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4\" (UID: \"c3d179f1-c7e0-4340-a286-2efbbc44ed53\") " pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4" Nov 02 01:35:50.829465 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:50.823928 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3d179f1-c7e0-4340-a286-2efbbc44ed53-istio-data\") pod \"e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4\" (UID: \"c3d179f1-c7e0-4340-a286-2efbbc44ed53\") " pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4" Nov 02 01:35:50.829465 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:50.824188 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl9pj\" (UniqueName: \"kubernetes.io/projected/c3d179f1-c7e0-4340-a286-2efbbc44ed53-kube-api-access-sl9pj\") pod \"e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4\" (UID: \"c3d179f1-c7e0-4340-a286-2efbbc44ed53\") " pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4" Nov 02 01:35:50.855399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:50.830421 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3d179f1-c7e0-4340-a286-2efbbc44ed53-workload-certs\") pod \"e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4\" (UID: \"c3d179f1-c7e0-4340-a286-2efbbc44ed53\") " pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4" Nov 02 01:35:50.934145 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:50.856082 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3d179f1-c7e0-4340-a286-2efbbc44ed53-istio-data\") pod \"e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4\" (UID: \"c3d179f1-c7e0-4340-a286-2efbbc44ed53\") " pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4" Nov 02 01:35:50.934364 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:50.871997 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3d179f1-c7e0-4340-a286-2efbbc44ed53-istio-podinfo\") pod \"e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4\" (UID: \"c3d179f1-c7e0-4340-a286-2efbbc44ed53\") " pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4" Nov 02 01:35:50.934503 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:50.893952 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3d179f1-c7e0-4340-a286-2efbbc44ed53-istio-envoy\") pod \"e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4\" (UID: \"c3d179f1-c7e0-4340-a286-2efbbc44ed53\") " pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4" Nov 02 01:35:50.940472 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:50.905889 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3d179f1-c7e0-4340-a286-2efbbc44ed53-istio-token\") pod \"e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4\" (UID: \"c3d179f1-c7e0-4340-a286-2efbbc44ed53\") " pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4" Nov 02 01:35:50.940669 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:50.934068 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3d179f1-c7e0-4340-a286-2efbbc44ed53-credential-socket\") pod \"e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4\" (UID: \"c3d179f1-c7e0-4340-a286-2efbbc44ed53\") " pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4" Nov 02 01:35:51.116615 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:51.070484 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-50537f4f-67d6cc6b87-7w7v4"] Nov 02 01:35:51.223639 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:51.118742 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-50537f4f-67d6cc6b87-7w7v4" Nov 02 01:35:51.223639 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:51.132681 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-50537f4f\"" Nov 02 01:35:51.223639 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:51.146284 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl9pj\" (UniqueName: \"kubernetes.io/projected/c3d179f1-c7e0-4340-a286-2efbbc44ed53-kube-api-access-sl9pj\") pod \"e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4\" (UID: \"c3d179f1-c7e0-4340-a286-2efbbc44ed53\") " pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4" Nov 02 01:35:51.223639 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:51.202183 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-50537f4f-dockercfg-gx2q4\"" Nov 02 01:35:51.223639 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:51.204436 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-50537f4f-67d6cc6b87-7w7v4"] Nov 02 01:35:51.223639 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:51.204773 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4" Nov 02 01:35:51.284665 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:51.277326 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e387f69-ed5c-449a-9ba5-8e03a04b235a-plugins-config-volume\") pod \"payload-processing-e2e-mt-50537f4f-67d6cc6b87-7w7v4\" (UID: \"6e387f69-ed5c-449a-9ba5-8e03a04b235a\") " pod="openshift-ingress/payload-processing-e2e-mt-50537f4f-67d6cc6b87-7w7v4" Nov 02 01:35:51.284665 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:51.277914 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7znw\" (UniqueName: \"kubernetes.io/projected/6e387f69-ed5c-449a-9ba5-8e03a04b235a-kube-api-access-l7znw\") pod \"payload-processing-e2e-mt-50537f4f-67d6cc6b87-7w7v4\" (UID: \"6e387f69-ed5c-449a-9ba5-8e03a04b235a\") " pod="openshift-ingress/payload-processing-e2e-mt-50537f4f-67d6cc6b87-7w7v4" Nov 02 01:35:51.380551 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:51.380067 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e387f69-ed5c-449a-9ba5-8e03a04b235a-plugins-config-volume\") pod \"payload-processing-e2e-mt-50537f4f-67d6cc6b87-7w7v4\" (UID: \"6e387f69-ed5c-449a-9ba5-8e03a04b235a\") " pod="openshift-ingress/payload-processing-e2e-mt-50537f4f-67d6cc6b87-7w7v4" Nov 02 01:35:51.381261 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:51.381053 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7znw\" (UniqueName: \"kubernetes.io/projected/6e387f69-ed5c-449a-9ba5-8e03a04b235a-kube-api-access-l7znw\") pod \"payload-processing-e2e-mt-50537f4f-67d6cc6b87-7w7v4\" (UID: \"6e387f69-ed5c-449a-9ba5-8e03a04b235a\") " pod="openshift-ingress/payload-processing-e2e-mt-50537f4f-67d6cc6b87-7w7v4" Nov 02 01:35:51.389465 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:51.389343 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e387f69-ed5c-449a-9ba5-8e03a04b235a-plugins-config-volume\") pod \"payload-processing-e2e-mt-50537f4f-67d6cc6b87-7w7v4\" (UID: \"6e387f69-ed5c-449a-9ba5-8e03a04b235a\") " pod="openshift-ingress/payload-processing-e2e-mt-50537f4f-67d6cc6b87-7w7v4" Nov 02 01:35:51.455193 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:51.455102 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7znw\" (UniqueName: \"kubernetes.io/projected/6e387f69-ed5c-449a-9ba5-8e03a04b235a-kube-api-access-l7znw\") pod \"payload-processing-e2e-mt-50537f4f-67d6cc6b87-7w7v4\" (UID: \"6e387f69-ed5c-449a-9ba5-8e03a04b235a\") " pod="openshift-ingress/payload-processing-e2e-mt-50537f4f-67d6cc6b87-7w7v4" Nov 02 01:35:51.620446 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:51.620277 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-50537f4f-67d6cc6b87-7w7v4" Nov 02 01:35:51.785683 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:51.782782 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4"] Nov 02 01:35:51.793238 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:51.793201 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:35:51.793404 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:51.793273 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:35:51.793404 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:51.793298 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:35:51.909724 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:51.909689 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4" event={"ID":"c3d179f1-c7e0-4340-a286-2efbbc44ed53","Type":"ContainerStarted","Data":"88900bf442a24ca8fdbb3b9c414e93bb9a8e230cc86eabf5c29c885c6e904ae2"} Nov 02 01:35:51.960071 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:51.958639 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-50537f4f-67d6cc6b87-7w7v4"] Nov 02 01:35:52.919351 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:52.919313 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-50537f4f-67d6cc6b87-7w7v4" event={"ID":"6e387f69-ed5c-449a-9ba5-8e03a04b235a","Type":"ContainerStarted","Data":"8cd30d6081c67e1aaed4ca03165ed4b107615abcc5842066617aa70c153bc0b0"} Nov 02 01:35:52.919868 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:52.919395 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-50537f4f-67d6cc6b87-7w7v4" event={"ID":"6e387f69-ed5c-449a-9ba5-8e03a04b235a","Type":"ContainerStarted","Data":"4d2b373f50d019271bfe8ef5797c7619c5ffa5885c188f01cb9a296a76ac24f9"} Nov 02 01:35:52.919868 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:52.919426 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-50537f4f-67d6cc6b87-7w7v4" Nov 02 01:35:52.921545 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:52.921515 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4" event={"ID":"c3d179f1-c7e0-4340-a286-2efbbc44ed53","Type":"ContainerStarted","Data":"662874da8f50b0842d9938672194dc31d4755f470d32aaa4db1518c240e7bfa5"} Nov 02 01:35:52.942323 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:52.942266 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-50537f4f-67d6cc6b87-7w7v4" podStartSLOduration=2.942251413 podStartE2EDuration="2.942251413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:35:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:35:52.936743033 +0000 UTC m=+1257.557748246" watchObservedRunningTime="2026-11-02 01:35:52.942251413 +0000 UTC m=+1257.563256624" Nov 02 01:35:52.961087 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:52.961036 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4" podStartSLOduration=2.961021006 podStartE2EDuration="2.961021006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:35:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:35:52.960493998 +0000 UTC m=+1257.581499215" watchObservedRunningTime="2026-11-02 01:35:52.961021006 +0000 UTC m=+1257.582026217" Nov 02 01:35:53.229755 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:53.229667 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4" Nov 02 01:35:53.235285 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:53.235256 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4" Nov 02 01:35:53.670293 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:53.670265 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-ffe5c1-579789c787-hz68t" Nov 02 01:35:53.929872 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:53.929792 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4" Nov 02 01:35:53.930683 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:53.930664 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4" Nov 02 01:35:56.060396 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:56.057708 4427 scope.go:122] "RemoveContainer" containerID="9a68797386cd091dff4b516815baa485f3cc08184f57cb45711379978cdad27c" Nov 02 01:35:56.164205 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:56.163830 4427 scope.go:122] "RemoveContainer" containerID="46a3de4207a3cb4ff259ece81dbfb31bfa1e20089e4fcb9174499515c59c7fec" Nov 02 01:35:56.897441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:56.897396 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b34fc0-6cd9f9c5fc-9n8zq"] Nov 02 01:35:56.903865 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:56.903834 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b34fc0-6cd9f9c5fc-9n8zq" Nov 02 01:35:56.908568 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:56.908537 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-b34fc0\"" Nov 02 01:35:56.921037 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:56.920998 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e58e874-db28-4a0e-8b46-368b6d4d0960-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b34fc0-6cd9f9c5fc-9n8zq\" (UID: \"1e58e874-db28-4a0e-8b46-368b6d4d0960\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b34fc0-6cd9f9c5fc-9n8zq" Nov 02 01:35:56.921391 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:56.921347 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nghgl\" (UniqueName: \"kubernetes.io/projected/1e58e874-db28-4a0e-8b46-368b6d4d0960-kube-api-access-nghgl\") pod \"payload-processing-e2e-inf-a-b34fc0-6cd9f9c5fc-9n8zq\" (UID: \"1e58e874-db28-4a0e-8b46-368b6d4d0960\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b34fc0-6cd9f9c5fc-9n8zq" Nov 02 01:35:56.926351 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:56.926313 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b34fc0-6cd9f9c5fc-9n8zq"] Nov 02 01:35:56.927413 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:56.927390 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-b34fc0-dockercfg-nfcnz\"" Nov 02 01:35:56.943936 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:56.943900 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b34fc0-5b786c7bdc-8lrhc"] Nov 02 01:35:56.956455 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:56.956360 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b34fc0-5b786c7bdc-8lrhc"] Nov 02 01:35:56.956624 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:56.956526 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b34fc0-5b786c7bdc-8lrhc" Nov 02 01:35:57.022722 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:57.022689 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp254\" (UniqueName: \"kubernetes.io/projected/d10ccd31-3d14-498e-bf7a-0b3034576528-kube-api-access-rp254\") pod \"payload-pre-processing-e2e-inf-a-b34fc0-5b786c7bdc-8lrhc\" (UID: \"d10ccd31-3d14-498e-bf7a-0b3034576528\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b34fc0-5b786c7bdc-8lrhc" Nov 02 01:35:57.022908 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:57.022820 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d10ccd31-3d14-498e-bf7a-0b3034576528-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b34fc0-5b786c7bdc-8lrhc\" (UID: \"d10ccd31-3d14-498e-bf7a-0b3034576528\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b34fc0-5b786c7bdc-8lrhc" Nov 02 01:35:57.022908 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:57.022897 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e58e874-db28-4a0e-8b46-368b6d4d0960-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b34fc0-6cd9f9c5fc-9n8zq\" (UID: \"1e58e874-db28-4a0e-8b46-368b6d4d0960\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b34fc0-6cd9f9c5fc-9n8zq" Nov 02 01:35:57.023015 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:57.023003 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nghgl\" (UniqueName: \"kubernetes.io/projected/1e58e874-db28-4a0e-8b46-368b6d4d0960-kube-api-access-nghgl\") pod \"payload-processing-e2e-inf-a-b34fc0-6cd9f9c5fc-9n8zq\" (UID: \"1e58e874-db28-4a0e-8b46-368b6d4d0960\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b34fc0-6cd9f9c5fc-9n8zq" Nov 02 01:35:57.023642 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:57.023621 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e58e874-db28-4a0e-8b46-368b6d4d0960-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b34fc0-6cd9f9c5fc-9n8zq\" (UID: \"1e58e874-db28-4a0e-8b46-368b6d4d0960\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b34fc0-6cd9f9c5fc-9n8zq" Nov 02 01:35:57.036160 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:57.036137 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nghgl\" (UniqueName: \"kubernetes.io/projected/1e58e874-db28-4a0e-8b46-368b6d4d0960-kube-api-access-nghgl\") pod \"payload-processing-e2e-inf-a-b34fc0-6cd9f9c5fc-9n8zq\" (UID: \"1e58e874-db28-4a0e-8b46-368b6d4d0960\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b34fc0-6cd9f9c5fc-9n8zq" Nov 02 01:35:57.124431 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:57.124392 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp254\" (UniqueName: \"kubernetes.io/projected/d10ccd31-3d14-498e-bf7a-0b3034576528-kube-api-access-rp254\") pod \"payload-pre-processing-e2e-inf-a-b34fc0-5b786c7bdc-8lrhc\" (UID: \"d10ccd31-3d14-498e-bf7a-0b3034576528\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b34fc0-5b786c7bdc-8lrhc" Nov 02 01:35:57.124936 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:57.124496 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d10ccd31-3d14-498e-bf7a-0b3034576528-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b34fc0-5b786c7bdc-8lrhc\" (UID: \"d10ccd31-3d14-498e-bf7a-0b3034576528\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b34fc0-5b786c7bdc-8lrhc" Nov 02 01:35:57.125263 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:57.125241 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d10ccd31-3d14-498e-bf7a-0b3034576528-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b34fc0-5b786c7bdc-8lrhc\" (UID: \"d10ccd31-3d14-498e-bf7a-0b3034576528\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b34fc0-5b786c7bdc-8lrhc" Nov 02 01:35:57.137617 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:57.137583 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp254\" (UniqueName: \"kubernetes.io/projected/d10ccd31-3d14-498e-bf7a-0b3034576528-kube-api-access-rp254\") pod \"payload-pre-processing-e2e-inf-a-b34fc0-5b786c7bdc-8lrhc\" (UID: \"d10ccd31-3d14-498e-bf7a-0b3034576528\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b34fc0-5b786c7bdc-8lrhc" Nov 02 01:35:57.237452 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:57.236922 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b34fc0-6cd9f9c5fc-9n8zq" Nov 02 01:35:57.335805 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:57.335773 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b34fc0-5b786c7bdc-8lrhc" Nov 02 01:35:57.820730 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:35:57.820663 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e58e874_db28_4a0e_8b46_368b6d4d0960.slice/crio-1ee3dea6889ca41a43a16ee6c0e42d09daac35055dc0e032dab9d5392018c6a5 WatchSource:0}: Error finding container 1ee3dea6889ca41a43a16ee6c0e42d09daac35055dc0e032dab9d5392018c6a5: Status 404 returned error can't find the container with id 1ee3dea6889ca41a43a16ee6c0e42d09daac35055dc0e032dab9d5392018c6a5 Nov 02 01:35:57.820837 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:57.820792 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b34fc0-6cd9f9c5fc-9n8zq"] Nov 02 01:35:57.856572 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:57.856536 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b34fc0-5b786c7bdc-8lrhc"] Nov 02 01:35:58.076072 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:58.075979 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b34fc0-5b786c7bdc-8lrhc" event={"ID":"d10ccd31-3d14-498e-bf7a-0b3034576528","Type":"ContainerStarted","Data":"ba8d5bfd013544907336604267b6db353fb410700c119045e616171898fb616a"} Nov 02 01:35:58.076072 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:58.076047 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b34fc0-5b786c7bdc-8lrhc" event={"ID":"d10ccd31-3d14-498e-bf7a-0b3034576528","Type":"ContainerStarted","Data":"46b4e86e34bc0a6a6c0df5509b62f3cc63e2cd42351bfb89862d73618fe2a340"} Nov 02 01:35:58.076348 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:58.076090 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b34fc0-5b786c7bdc-8lrhc" Nov 02 01:35:58.080089 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:58.080059 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b34fc0-6cd9f9c5fc-9n8zq" event={"ID":"1e58e874-db28-4a0e-8b46-368b6d4d0960","Type":"ContainerStarted","Data":"d65fa69dbf28c35abecfbb9bcfc5765dc93643496377c7228266dd0edfc22ec0"} Nov 02 01:35:58.080278 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:58.080259 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b34fc0-6cd9f9c5fc-9n8zq" event={"ID":"1e58e874-db28-4a0e-8b46-368b6d4d0960","Type":"ContainerStarted","Data":"1ee3dea6889ca41a43a16ee6c0e42d09daac35055dc0e032dab9d5392018c6a5"} Nov 02 01:35:58.080534 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:58.080515 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-b34fc0-6cd9f9c5fc-9n8zq" Nov 02 01:35:58.098860 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:58.098782 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b34fc0-5b786c7bdc-8lrhc" podStartSLOduration=2.098766938 podStartE2EDuration="2.098766938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:35:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:35:58.09723934 +0000 UTC m=+1262.718244550" watchObservedRunningTime="2026-11-02 01:35:58.098766938 +0000 UTC m=+1262.719772158" Nov 02 01:35:58.123008 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:35:58.122952 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-b34fc0-6cd9f9c5fc-9n8zq" podStartSLOduration=2.122938291 podStartE2EDuration="2.122938291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:35:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:35:58.121039631 +0000 UTC m=+1262.742044868" watchObservedRunningTime="2026-11-02 01:35:58.122938291 +0000 UTC m=+1262.743943501" Nov 02 01:36:01.671896 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:36:01.671808 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:36:02.249821 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:02.249769 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" podUID="366c78ee-4aaf-4b7c-84b4-17028538cb1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:36:02.307537 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:02.307477 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" podUID="7085c0ab-7ee3-4665-abad-2977c25f68ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:36:02.316702 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:02.316656 4427 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" podUID="d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:36:03.771784 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:03.769510 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" podUID="706df1bd-19cf-4ab9-9aaa-3c12c97ccf45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:36:04.070110 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:04.070018 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-50537f4f-67d6cc6b87-7w7v4" Nov 02 01:36:04.253515 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:04.253466 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" podUID="366c78ee-4aaf-4b7c-84b4-17028538cb1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:36:04.348742 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:04.343472 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" podUID="ea61971c-5a72-4609-b5a0-58a74b324bb7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 01:36:08.698018 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:08.697972 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2589b2-5b4dc6cc9f-nmtkc"] Nov 02 01:36:08.741503 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:08.741457 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2589b2-5b4dc6cc9f-nmtkc"] Nov 02 01:36:08.741701 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:08.741625 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2589b2-5b4dc6cc9f-nmtkc" Nov 02 01:36:08.761350 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:08.761324 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-2589b2\"" Nov 02 01:36:08.769859 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:08.769835 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-2589b2-dockercfg-z8rc6\"" Nov 02 01:36:08.945026 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:08.944986 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-669b8\" (UniqueName: \"kubernetes.io/projected/a097a1f1-7f03-479d-a65a-90f8e542d06b-kube-api-access-669b8\") pod \"payload-pre-processing-e2e-inf-b-2589b2-5b4dc6cc9f-nmtkc\" (UID: \"a097a1f1-7f03-479d-a65a-90f8e542d06b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2589b2-5b4dc6cc9f-nmtkc" Nov 02 01:36:08.945534 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:08.945508 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a097a1f1-7f03-479d-a65a-90f8e542d06b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2589b2-5b4dc6cc9f-nmtkc\" (UID: \"a097a1f1-7f03-479d-a65a-90f8e542d06b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2589b2-5b4dc6cc9f-nmtkc" Nov 02 01:36:09.052702 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:09.052591 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a097a1f1-7f03-479d-a65a-90f8e542d06b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2589b2-5b4dc6cc9f-nmtkc\" (UID: \"a097a1f1-7f03-479d-a65a-90f8e542d06b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2589b2-5b4dc6cc9f-nmtkc" Nov 02 01:36:09.054221 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:09.054193 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-669b8\" (UniqueName: \"kubernetes.io/projected/a097a1f1-7f03-479d-a65a-90f8e542d06b-kube-api-access-669b8\") pod \"payload-pre-processing-e2e-inf-b-2589b2-5b4dc6cc9f-nmtkc\" (UID: \"a097a1f1-7f03-479d-a65a-90f8e542d06b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2589b2-5b4dc6cc9f-nmtkc" Nov 02 01:36:09.056388 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:09.053793 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a097a1f1-7f03-479d-a65a-90f8e542d06b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2589b2-5b4dc6cc9f-nmtkc\" (UID: \"a097a1f1-7f03-479d-a65a-90f8e542d06b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2589b2-5b4dc6cc9f-nmtkc" Nov 02 01:36:09.143431 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:09.140845 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b34fc0-5b786c7bdc-8lrhc" Nov 02 01:36:09.190429 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:09.176556 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-b34fc0-6cd9f9c5fc-9n8zq" Nov 02 01:36:09.197909 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:09.197315 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-669b8\" (UniqueName: \"kubernetes.io/projected/a097a1f1-7f03-479d-a65a-90f8e542d06b-kube-api-access-669b8\") pod \"payload-pre-processing-e2e-inf-b-2589b2-5b4dc6cc9f-nmtkc\" (UID: \"a097a1f1-7f03-479d-a65a-90f8e542d06b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2589b2-5b4dc6cc9f-nmtkc" Nov 02 01:36:09.402230 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:09.402192 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2589b2-5b4dc6cc9f-nmtkc" Nov 02 01:36:09.766919 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:09.766883 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2589b2-5b4dc6cc9f-nmtkc"] Nov 02 01:36:09.770271 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:36:09.769694 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda097a1f1_7f03_479d_a65a_90f8e542d06b.slice/crio-b13291f9539f24f740c82745c97e264a7139399ba3aa1eeac3bc8222cd545d07 WatchSource:0}: Error finding container b13291f9539f24f740c82745c97e264a7139399ba3aa1eeac3bc8222cd545d07: Status 404 returned error can't find the container with id b13291f9539f24f740c82745c97e264a7139399ba3aa1eeac3bc8222cd545d07 Nov 02 01:36:09.932708 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:09.932673 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm"] Nov 02 01:36:09.932917 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:09.932893 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm" podUID="119ffe15-7bac-418f-a288-52e1e61203ed" containerName="istio-proxy" containerID="cri-o://25e6f4621b2abdb4c7cbbc62df36ef508ae58c2545915107fc4a46092e22aaf0" gracePeriod=30 Nov 02 01:36:10.012781 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:10.012668 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2589b2-5b4dc6cc9f-nmtkc" event={"ID":"a097a1f1-7f03-479d-a65a-90f8e542d06b","Type":"ContainerStarted","Data":"b99b75d2d5d0e26802f6fa73863ea110cfe140e58ceca80294d507733ca008e8"} Nov 02 01:36:10.013091 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:10.013071 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2589b2-5b4dc6cc9f-nmtkc" Nov 02 01:36:10.013177 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:10.013122 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2589b2-5b4dc6cc9f-nmtkc" event={"ID":"a097a1f1-7f03-479d-a65a-90f8e542d06b","Type":"ContainerStarted","Data":"b13291f9539f24f740c82745c97e264a7139399ba3aa1eeac3bc8222cd545d07"} Nov 02 01:36:10.033957 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:10.033892 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2589b2-5b4dc6cc9f-nmtkc" podStartSLOduration=2.033871437 podStartE2EDuration="2.033871437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:36:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:36:10.031644167 +0000 UTC m=+1274.652649390" watchObservedRunningTime="2026-11-02 01:36:10.033871437 +0000 UTC m=+1274.654876649" Nov 02 01:36:10.131913 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:10.131882 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2"] Nov 02 01:36:10.132106 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:10.132081 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2" podUID="165f361d-6b06-4c6c-8598-27c4b522b2e2" containerName="istio-proxy" containerID="cri-o://a931dca1e3a0c74d664fbab0dc4d3377ef5d222942da09c5888c3ba48bab3be2" gracePeriod=30 Nov 02 01:36:11.021933 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:11.021881 4427 generic.go:356] "Generic (PLEG): container finished" podID="e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0" containerID="db7574e374b5d799907ecd4f3b953b8d7c2afccfa515149ab1a326ea05191cae" exitCode=0 Nov 02 01:36:11.022454 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:11.021941 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l" event={"ID":"e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0","Type":"ContainerDied","Data":"db7574e374b5d799907ecd4f3b953b8d7c2afccfa515149ab1a326ea05191cae"} Nov 02 01:36:11.022454 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:11.022450 4427 scope.go:122] "RemoveContainer" containerID="db7574e374b5d799907ecd4f3b953b8d7c2afccfa515149ab1a326ea05191cae" Nov 02 01:36:11.024170 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:11.024140 4427 generic.go:356] "Generic (PLEG): container finished" podID="8f9b0669-291d-43fa-9511-d74948b03684" containerID="1a94ae972be836c0ad68ee977315577de6860497757ed3ef245783a084447f50" exitCode=0 Nov 02 01:36:11.024289 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:11.024212 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp" event={"ID":"8f9b0669-291d-43fa-9511-d74948b03684","Type":"ContainerDied","Data":"1a94ae972be836c0ad68ee977315577de6860497757ed3ef245783a084447f50"} Nov 02 01:36:11.024289 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:11.024246 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:36:11.024438 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:11.024358 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:36:11.024438 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:11.024429 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:36:11.024809 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:11.024788 4427 scope.go:122] "RemoveContainer" containerID="1a94ae972be836c0ad68ee977315577de6860497757ed3ef245783a084447f50" Nov 02 01:36:11.026393 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:11.026351 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:36:11.026493 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:11.026441 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:36:11.026556 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:11.026494 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:36:12.032755 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:12.032720 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp" event={"ID":"8f9b0669-291d-43fa-9511-d74948b03684","Type":"ContainerStarted","Data":"f3bbaed735d3a00f7bdd6497f8dc6c54770bb6cbf4923ddb3073775664a538eb"} Nov 02 01:36:12.036307 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:12.036280 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l" event={"ID":"e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0","Type":"ContainerStarted","Data":"5ac18ab0bce64b05e7e0e8b3a8a9c47fd50e9a7cf9d7e587aef100e5b71eeb5a"} Nov 02 01:36:12.369407 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:12.368744 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp" Nov 02 01:36:12.390401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:12.383520 4427 patch_prober.go:29] interesting pod/e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 02 01:36:12.390401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:12.383584 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp" podUID="8f9b0669-291d-43fa-9511-d74948b03684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 02 01:36:12.493400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:12.492962 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l" Nov 02 01:36:12.493400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:12.493111 4427 patch_prober.go:29] interesting pod/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 02 01:36:12.493400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:12.493155 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l" podUID="e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 02 01:36:13.381005 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:13.366876 4427 patch_prober.go:29] interesting pod/e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 02 01:36:13.381005 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:13.366950 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp" podUID="8f9b0669-291d-43fa-9511-d74948b03684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 02 01:36:13.508401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:13.501952 4427 patch_prober.go:29] interesting pod/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 02 01:36:13.508401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:13.502011 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l" podUID="e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 02 01:36:14.383850 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:14.383808 4427 patch_prober.go:29] interesting pod/e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 02 01:36:14.404648 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:14.404607 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp" podUID="8f9b0669-291d-43fa-9511-d74948b03684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 02 01:36:14.492856 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:14.492814 4427 patch_prober.go:29] interesting pod/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 02 01:36:14.508404 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:14.498462 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l" podUID="e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 02 01:36:15.400394 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:15.399776 4427 patch_prober.go:29] interesting pod/e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 02 01:36:15.400394 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:15.399834 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp" podUID="8f9b0669-291d-43fa-9511-d74948b03684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 02 01:36:15.492300 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:15.491797 4427 patch_prober.go:29] interesting pod/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 02 01:36:15.492300 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:15.491901 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l" podUID="e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 02 01:36:15.804159 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:15.803668 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm" Nov 02 01:36:15.976837 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:15.974553 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/119ffe15-7bac-418f-a288-52e1e61203ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/119ffe15-7bac-418f-a288-52e1e61203ed-istio-podinfo\") pod \"119ffe15-7bac-418f-a288-52e1e61203ed\" (UID: \"119ffe15-7bac-418f-a288-52e1e61203ed\") " Nov 02 01:36:15.976837 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:15.974671 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/119ffe15-7bac-418f-a288-52e1e61203ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/119ffe15-7bac-418f-a288-52e1e61203ed-workload-socket\") pod \"119ffe15-7bac-418f-a288-52e1e61203ed\" (UID: \"119ffe15-7bac-418f-a288-52e1e61203ed\") " Nov 02 01:36:16.043054 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.041478 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/119ffe15-7bac-418f-a288-52e1e61203ed-workload-socket" pod "119ffe15-7bac-418f-a288-52e1e61203ed" (UID: "119ffe15-7bac-418f-a288-52e1e61203ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:36:16.043054 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.041586 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/119ffe15-7bac-418f-a288-52e1e61203ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/119ffe15-7bac-418f-a288-52e1e61203ed-workload-certs\") pod \"119ffe15-7bac-418f-a288-52e1e61203ed\" (UID: \"119ffe15-7bac-418f-a288-52e1e61203ed\") " Nov 02 01:36:16.043054 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.041710 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/119ffe15-7bac-418f-a288-52e1e61203ed-kube-api-access-4sm92\" (UniqueName: \"kubernetes.io/projected/119ffe15-7bac-418f-a288-52e1e61203ed-kube-api-access-4sm92\") pod \"119ffe15-7bac-418f-a288-52e1e61203ed\" (UID: \"119ffe15-7bac-418f-a288-52e1e61203ed\") " Nov 02 01:36:16.043054 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.041764 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/119ffe15-7bac-418f-a288-52e1e61203ed-istio-token\" (UniqueName: \"kubernetes.io/projected/119ffe15-7bac-418f-a288-52e1e61203ed-istio-token\") pod \"119ffe15-7bac-418f-a288-52e1e61203ed\" (UID: \"119ffe15-7bac-418f-a288-52e1e61203ed\") " Nov 02 01:36:16.043054 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.041812 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/119ffe15-7bac-418f-a288-52e1e61203ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/119ffe15-7bac-418f-a288-52e1e61203ed-credential-socket\") pod \"119ffe15-7bac-418f-a288-52e1e61203ed\" (UID: \"119ffe15-7bac-418f-a288-52e1e61203ed\") " Nov 02 01:36:16.043054 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.041955 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/119ffe15-7bac-418f-a288-52e1e61203ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/119ffe15-7bac-418f-a288-52e1e61203ed-istio-data\") pod \"119ffe15-7bac-418f-a288-52e1e61203ed\" (UID: \"119ffe15-7bac-418f-a288-52e1e61203ed\") " Nov 02 01:36:16.043054 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.042049 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/119ffe15-7bac-418f-a288-52e1e61203ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/119ffe15-7bac-418f-a288-52e1e61203ed-istiod-ca-cert\") pod \"119ffe15-7bac-418f-a288-52e1e61203ed\" (UID: \"119ffe15-7bac-418f-a288-52e1e61203ed\") " Nov 02 01:36:16.043054 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.042146 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/119ffe15-7bac-418f-a288-52e1e61203ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/119ffe15-7bac-418f-a288-52e1e61203ed-istio-envoy\") pod \"119ffe15-7bac-418f-a288-52e1e61203ed\" (UID: \"119ffe15-7bac-418f-a288-52e1e61203ed\") " Nov 02 01:36:16.043635 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.043342 4427 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/119ffe15-7bac-418f-a288-52e1e61203ed-workload-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:16.126543 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.125407 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/119ffe15-7bac-418f-a288-52e1e61203ed-istio-data" pod "119ffe15-7bac-418f-a288-52e1e61203ed" (UID: "119ffe15-7bac-418f-a288-52e1e61203ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:36:16.126543 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.126477 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/119ffe15-7bac-418f-a288-52e1e61203ed-credential-socket" pod "119ffe15-7bac-418f-a288-52e1e61203ed" (UID: "119ffe15-7bac-418f-a288-52e1e61203ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:36:16.149437 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.133081 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/119ffe15-7bac-418f-a288-52e1e61203ed-istiod-ca-cert" pod "119ffe15-7bac-418f-a288-52e1e61203ed" (UID: "119ffe15-7bac-418f-a288-52e1e61203ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:36:16.199014 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.198069 4427 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/119ffe15-7bac-418f-a288-52e1e61203ed-credential-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:16.199014 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.198103 4427 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/119ffe15-7bac-418f-a288-52e1e61203ed-istio-data\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:16.199014 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.198122 4427 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/119ffe15-7bac-418f-a288-52e1e61203ed-istiod-ca-cert\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:16.248517 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.245705 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/119ffe15-7bac-418f-a288-52e1e61203ed-workload-certs" pod "119ffe15-7bac-418f-a288-52e1e61203ed" (UID: "119ffe15-7bac-418f-a288-52e1e61203ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:36:16.252622 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.249301 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/119ffe15-7bac-418f-a288-52e1e61203ed-istio-envoy" pod "119ffe15-7bac-418f-a288-52e1e61203ed" (UID: "119ffe15-7bac-418f-a288-52e1e61203ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:36:16.252622 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.249478 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/119ffe15-7bac-418f-a288-52e1e61203ed-kube-api-access-4sm92" pod "119ffe15-7bac-418f-a288-52e1e61203ed" (UID: "119ffe15-7bac-418f-a288-52e1e61203ed"). InnerVolumeSpecName "kube-api-access-4sm92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:36:16.252622 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.249553 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/119ffe15-7bac-418f-a288-52e1e61203ed-istio-token" pod "119ffe15-7bac-418f-a288-52e1e61203ed" (UID: "119ffe15-7bac-418f-a288-52e1e61203ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:36:16.264484 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.264447 4427 generic.go:356] "Generic (PLEG): container finished" podID="165f361d-6b06-4c6c-8598-27c4b522b2e2" containerID="a931dca1e3a0c74d664fbab0dc4d3377ef5d222942da09c5888c3ba48bab3be2" exitCode=0 Nov 02 01:36:16.264799 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.264775 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2" event={"ID":"165f361d-6b06-4c6c-8598-27c4b522b2e2","Type":"ContainerDied","Data":"a931dca1e3a0c74d664fbab0dc4d3377ef5d222942da09c5888c3ba48bab3be2"} Nov 02 01:36:16.319402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.300873 4427 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/119ffe15-7bac-418f-a288-52e1e61203ed-istio-envoy\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:16.319402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.300908 4427 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/119ffe15-7bac-418f-a288-52e1e61203ed-workload-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:16.319402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.300926 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sm92\" (UniqueName: \"kubernetes.io/projected/119ffe15-7bac-418f-a288-52e1e61203ed-kube-api-access-4sm92\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:16.319402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.300941 4427 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/119ffe15-7bac-418f-a288-52e1e61203ed-istio-token\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:16.339045 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.338971 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/119ffe15-7bac-418f-a288-52e1e61203ed-istio-podinfo" pod "119ffe15-7bac-418f-a288-52e1e61203ed" (UID: "119ffe15-7bac-418f-a288-52e1e61203ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:36:16.342785 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.342737 4427 generic.go:356] "Generic (PLEG): container finished" podID="119ffe15-7bac-418f-a288-52e1e61203ed" containerID="25e6f4621b2abdb4c7cbbc62df36ef508ae58c2545915107fc4a46092e22aaf0" exitCode=0 Nov 02 01:36:16.343243 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.343200 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm" event={"ID":"119ffe15-7bac-418f-a288-52e1e61203ed","Type":"ContainerDied","Data":"25e6f4621b2abdb4c7cbbc62df36ef508ae58c2545915107fc4a46092e22aaf0"} Nov 02 01:36:16.343468 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.343396 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm" event={"ID":"119ffe15-7bac-418f-a288-52e1e61203ed","Type":"ContainerDied","Data":"adff0d4fce22cf6acc9a0f24fd77655f6ef15905e6d60d812bd2f5593395ecb1"} Nov 02 01:36:16.343468 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.343433 4427 scope.go:122] "RemoveContainer" containerID="25e6f4621b2abdb4c7cbbc62df36ef508ae58c2545915107fc4a46092e22aaf0" Nov 02 01:36:16.343860 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.343803 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm" Nov 02 01:36:16.416138 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.416046 4427 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/119ffe15-7bac-418f-a288-52e1e61203ed-istio-podinfo\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:16.419920 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.419773 4427 patch_prober.go:29] interesting pod/e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 02 01:36:16.420871 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.420842 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp" podUID="8f9b0669-291d-43fa-9511-d74948b03684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 02 01:36:16.430624 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.428454 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2" Nov 02 01:36:16.442208 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.441863 4427 scope.go:122] "RemoveContainer" containerID="25e6f4621b2abdb4c7cbbc62df36ef508ae58c2545915107fc4a46092e22aaf0" Nov 02 01:36:16.445061 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:36:16.445008 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25e6f4621b2abdb4c7cbbc62df36ef508ae58c2545915107fc4a46092e22aaf0\": container with ID starting with 25e6f4621b2abdb4c7cbbc62df36ef508ae58c2545915107fc4a46092e22aaf0 not found: ID does not exist" containerID="25e6f4621b2abdb4c7cbbc62df36ef508ae58c2545915107fc4a46092e22aaf0" Nov 02 01:36:16.445179 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.445078 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25e6f4621b2abdb4c7cbbc62df36ef508ae58c2545915107fc4a46092e22aaf0"} err="failed to get container status \"25e6f4621b2abdb4c7cbbc62df36ef508ae58c2545915107fc4a46092e22aaf0\": rpc error: code = NotFound desc = could not find container \"25e6f4621b2abdb4c7cbbc62df36ef508ae58c2545915107fc4a46092e22aaf0\": container with ID starting with 25e6f4621b2abdb4c7cbbc62df36ef508ae58c2545915107fc4a46092e22aaf0 not found: ID does not exist" Nov 02 01:36:16.502539 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.502495 4427 patch_prober.go:29] interesting pod/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 02 01:36:16.502813 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.502780 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l" podUID="e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 02 01:36:16.520461 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.520425 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/165f361d-6b06-4c6c-8598-27c4b522b2e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/165f361d-6b06-4c6c-8598-27c4b522b2e2-istio-podinfo\") pod \"165f361d-6b06-4c6c-8598-27c4b522b2e2\" (UID: \"165f361d-6b06-4c6c-8598-27c4b522b2e2\") " Nov 02 01:36:16.521724 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.521701 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/165f361d-6b06-4c6c-8598-27c4b522b2e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/165f361d-6b06-4c6c-8598-27c4b522b2e2-workload-socket\") pod \"165f361d-6b06-4c6c-8598-27c4b522b2e2\" (UID: \"165f361d-6b06-4c6c-8598-27c4b522b2e2\") " Nov 02 01:36:16.522249 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.522136 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/165f361d-6b06-4c6c-8598-27c4b522b2e2-workload-socket" pod "165f361d-6b06-4c6c-8598-27c4b522b2e2" (UID: "165f361d-6b06-4c6c-8598-27c4b522b2e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:36:16.522422 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.522405 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/165f361d-6b06-4c6c-8598-27c4b522b2e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/165f361d-6b06-4c6c-8598-27c4b522b2e2-workload-certs\") pod \"165f361d-6b06-4c6c-8598-27c4b522b2e2\" (UID: \"165f361d-6b06-4c6c-8598-27c4b522b2e2\") " Nov 02 01:36:16.522884 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.522759 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/165f361d-6b06-4c6c-8598-27c4b522b2e2-workload-certs" pod "165f361d-6b06-4c6c-8598-27c4b522b2e2" (UID: "165f361d-6b06-4c6c-8598-27c4b522b2e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:36:16.523165 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.523147 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/165f361d-6b06-4c6c-8598-27c4b522b2e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/165f361d-6b06-4c6c-8598-27c4b522b2e2-istiod-ca-cert\") pod \"165f361d-6b06-4c6c-8598-27c4b522b2e2\" (UID: \"165f361d-6b06-4c6c-8598-27c4b522b2e2\") " Nov 02 01:36:16.523729 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.523621 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/165f361d-6b06-4c6c-8598-27c4b522b2e2-istiod-ca-cert" pod "165f361d-6b06-4c6c-8598-27c4b522b2e2" (UID: "165f361d-6b06-4c6c-8598-27c4b522b2e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:36:16.524112 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.523686 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/165f361d-6b06-4c6c-8598-27c4b522b2e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/165f361d-6b06-4c6c-8598-27c4b522b2e2-istio-envoy\") pod \"165f361d-6b06-4c6c-8598-27c4b522b2e2\" (UID: \"165f361d-6b06-4c6c-8598-27c4b522b2e2\") " Nov 02 01:36:16.524365 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.524347 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/165f361d-6b06-4c6c-8598-27c4b522b2e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/165f361d-6b06-4c6c-8598-27c4b522b2e2-istio-data\") pod \"165f361d-6b06-4c6c-8598-27c4b522b2e2\" (UID: \"165f361d-6b06-4c6c-8598-27c4b522b2e2\") " Nov 02 01:36:16.524552 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.524534 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/165f361d-6b06-4c6c-8598-27c4b522b2e2-kube-api-access-n94kc\" (UniqueName: \"kubernetes.io/projected/165f361d-6b06-4c6c-8598-27c4b522b2e2-kube-api-access-n94kc\") pod \"165f361d-6b06-4c6c-8598-27c4b522b2e2\" (UID: \"165f361d-6b06-4c6c-8598-27c4b522b2e2\") " Nov 02 01:36:16.525575 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.524736 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/165f361d-6b06-4c6c-8598-27c4b522b2e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/165f361d-6b06-4c6c-8598-27c4b522b2e2-credential-socket\") pod \"165f361d-6b06-4c6c-8598-27c4b522b2e2\" (UID: \"165f361d-6b06-4c6c-8598-27c4b522b2e2\") " Nov 02 01:36:16.526214 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.526196 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/165f361d-6b06-4c6c-8598-27c4b522b2e2-istio-token\" (UniqueName: \"kubernetes.io/projected/165f361d-6b06-4c6c-8598-27c4b522b2e2-istio-token\") pod \"165f361d-6b06-4c6c-8598-27c4b522b2e2\" (UID: \"165f361d-6b06-4c6c-8598-27c4b522b2e2\") " Nov 02 01:36:16.527161 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.525860 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/165f361d-6b06-4c6c-8598-27c4b522b2e2-istio-data" pod "165f361d-6b06-4c6c-8598-27c4b522b2e2" (UID: "165f361d-6b06-4c6c-8598-27c4b522b2e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:36:16.527161 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.526635 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/165f361d-6b06-4c6c-8598-27c4b522b2e2-credential-socket" pod "165f361d-6b06-4c6c-8598-27c4b522b2e2" (UID: "165f361d-6b06-4c6c-8598-27c4b522b2e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:36:16.527335 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.527317 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/165f361d-6b06-4c6c-8598-27c4b522b2e2-istio-envoy" pod "165f361d-6b06-4c6c-8598-27c4b522b2e2" (UID: "165f361d-6b06-4c6c-8598-27c4b522b2e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:36:16.527416 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.527342 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/165f361d-6b06-4c6c-8598-27c4b522b2e2-istio-podinfo" pod "165f361d-6b06-4c6c-8598-27c4b522b2e2" (UID: "165f361d-6b06-4c6c-8598-27c4b522b2e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:36:16.527821 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.527801 4427 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/165f361d-6b06-4c6c-8598-27c4b522b2e2-istiod-ca-cert\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:16.528119 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.528103 4427 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/165f361d-6b06-4c6c-8598-27c4b522b2e2-istio-envoy\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:16.528214 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.528202 4427 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/165f361d-6b06-4c6c-8598-27c4b522b2e2-istio-data\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:16.528340 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.528325 4427 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/165f361d-6b06-4c6c-8598-27c4b522b2e2-credential-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:16.528505 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.528487 4427 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/165f361d-6b06-4c6c-8598-27c4b522b2e2-istio-podinfo\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:16.528631 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.528617 4427 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/165f361d-6b06-4c6c-8598-27c4b522b2e2-workload-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:16.528734 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.528719 4427 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/165f361d-6b06-4c6c-8598-27c4b522b2e2-workload-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:16.528820 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.528344 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/165f361d-6b06-4c6c-8598-27c4b522b2e2-kube-api-access-n94kc" pod "165f361d-6b06-4c6c-8598-27c4b522b2e2" (UID: "165f361d-6b06-4c6c-8598-27c4b522b2e2"). InnerVolumeSpecName "kube-api-access-n94kc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:36:16.529542 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.529521 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/165f361d-6b06-4c6c-8598-27c4b522b2e2-istio-token" pod "165f361d-6b06-4c6c-8598-27c4b522b2e2" (UID: "165f361d-6b06-4c6c-8598-27c4b522b2e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:36:16.529655 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.529536 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm"] Nov 02 01:36:16.534061 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.534041 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-dbxsm"] Nov 02 01:36:16.652506 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.652464 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n94kc\" (UniqueName: \"kubernetes.io/projected/165f361d-6b06-4c6c-8598-27c4b522b2e2-kube-api-access-n94kc\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:16.652895 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:16.652861 4427 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/165f361d-6b06-4c6c-8598-27c4b522b2e2-istio-token\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:17.011521 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:17.011488 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp" Nov 02 01:36:17.354363 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:17.354279 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2" event={"ID":"165f361d-6b06-4c6c-8598-27c4b522b2e2","Type":"ContainerDied","Data":"81dcfffc8047cdcf530580f4d46323763bc51aff3f133ec1de8aeb6f15a2c248"} Nov 02 01:36:17.354363 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:17.354316 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2" Nov 02 01:36:17.354363 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:17.354349 4427 scope.go:122] "RemoveContainer" containerID="a931dca1e3a0c74d664fbab0dc4d3377ef5d222942da09c5888c3ba48bab3be2" Nov 02 01:36:17.388402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:17.367718 4427 patch_prober.go:29] interesting pod/e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 02 01:36:17.389253 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:17.389220 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp" podUID="8f9b0669-291d-43fa-9511-d74948b03684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 02 01:36:17.430874 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:17.430841 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2"] Nov 02 01:36:17.435298 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:17.435273 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-9b7p2"] Nov 02 01:36:17.520444 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:17.497704 4427 patch_prober.go:29] interesting pod/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 02 01:36:17.520444 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:17.497766 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l" podUID="e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 02 01:36:17.649431 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:17.648935 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="119ffe15-7bac-418f-a288-52e1e61203ed" path="/var/lib/kubelet/pods/119ffe15-7bac-418f-a288-52e1e61203ed/volumes" Nov 02 01:36:17.650446 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:17.649903 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="165f361d-6b06-4c6c-8598-27c4b522b2e2" path="/var/lib/kubelet/pods/165f361d-6b06-4c6c-8598-27c4b522b2e2/volumes" Nov 02 01:36:18.388923 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:18.388877 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp" Nov 02 01:36:18.391742 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:18.391711 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp" Nov 02 01:36:18.500449 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:18.500392 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l" Nov 02 01:36:18.501209 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:18.501185 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l" Nov 02 01:36:18.503129 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:18.503105 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l" Nov 02 01:36:21.029183 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:21.029155 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2589b2-5b4dc6cc9f-nmtkc" Nov 02 01:36:22.499457 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:22.499394 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-0bbeda/test-model-0bbeda-kserve-5cb56fbff9-lqp9j"] Nov 02 01:36:22.500455 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:22.500430 4427 cpu_manager.go:414] "containerMap: removing container" podUID="119ffe15-7bac-418f-a288-52e1e61203ed" containerName="istio-proxy" Nov 02 01:36:22.500455 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:22.500453 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="119ffe15-7bac-418f-a288-52e1e61203ed" containerName="istio-proxy" Nov 02 01:36:22.500663 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:22.500469 4427 cpu_manager.go:414] "containerMap: removing container" podUID="165f361d-6b06-4c6c-8598-27c4b522b2e2" containerName="istio-proxy" Nov 02 01:36:22.500663 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:22.500478 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="165f361d-6b06-4c6c-8598-27c4b522b2e2" containerName="istio-proxy" Nov 02 01:36:22.500759 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:22.500697 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="119ffe15-7bac-418f-a288-52e1e61203ed" containerName="istio-proxy" Nov 02 01:36:22.500759 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:22.500714 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="165f361d-6b06-4c6c-8598-27c4b522b2e2" containerName="istio-proxy" Nov 02 01:36:22.507198 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:22.507176 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0bbeda/test-model-0bbeda-kserve-5cb56fbff9-lqp9j" Nov 02 01:36:22.511586 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:22.511542 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-0bbeda\"/\"kube-root-ca.crt\"" Nov 02 01:36:22.511750 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:22.511733 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-0bbeda\"/\"default-dockercfg-d64sn\"" Nov 02 01:36:22.511916 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:22.511898 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-0bbeda\"/\"test-model-0bbeda-kserve-self-signed-certs\"" Nov 02 01:36:22.516683 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:22.516654 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-0bbeda/test-model-0bbeda-kserve-5cb56fbff9-lqp9j"] Nov 02 01:36:22.607996 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:22.607962 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-0bbeda\"/\"openshift-service-ca.crt\"" Nov 02 01:36:22.612981 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:22.612948 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b45e955d-0d37-44c2-96be-42dc39687913-tmp-dir\") pod \"test-model-0bbeda-kserve-5cb56fbff9-lqp9j\" (UID: \"b45e955d-0d37-44c2-96be-42dc39687913\") " pod="ai-tenant-e2e-inf-a-0bbeda/test-model-0bbeda-kserve-5cb56fbff9-lqp9j" Nov 02 01:36:22.613165 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:22.612999 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b45e955d-0d37-44c2-96be-42dc39687913-model-cache\") pod \"test-model-0bbeda-kserve-5cb56fbff9-lqp9j\" (UID: \"b45e955d-0d37-44c2-96be-42dc39687913\") " pod="ai-tenant-e2e-inf-a-0bbeda/test-model-0bbeda-kserve-5cb56fbff9-lqp9j" Nov 02 01:36:22.613165 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:22.613111 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6bpc\" (UniqueName: \"kubernetes.io/projected/b45e955d-0d37-44c2-96be-42dc39687913-kube-api-access-k6bpc\") pod \"test-model-0bbeda-kserve-5cb56fbff9-lqp9j\" (UID: \"b45e955d-0d37-44c2-96be-42dc39687913\") " pod="ai-tenant-e2e-inf-a-0bbeda/test-model-0bbeda-kserve-5cb56fbff9-lqp9j" Nov 02 01:36:22.613294 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:22.613166 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b45e955d-0d37-44c2-96be-42dc39687913-tls-certs\") pod \"test-model-0bbeda-kserve-5cb56fbff9-lqp9j\" (UID: \"b45e955d-0d37-44c2-96be-42dc39687913\") " pod="ai-tenant-e2e-inf-a-0bbeda/test-model-0bbeda-kserve-5cb56fbff9-lqp9j" Nov 02 01:36:22.613385 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:22.613357 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b45e955d-0d37-44c2-96be-42dc39687913-home\") pod \"test-model-0bbeda-kserve-5cb56fbff9-lqp9j\" (UID: \"b45e955d-0d37-44c2-96be-42dc39687913\") " pod="ai-tenant-e2e-inf-a-0bbeda/test-model-0bbeda-kserve-5cb56fbff9-lqp9j" Nov 02 01:36:22.613546 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:22.613514 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b45e955d-0d37-44c2-96be-42dc39687913-dshm\") pod \"test-model-0bbeda-kserve-5cb56fbff9-lqp9j\" (UID: \"b45e955d-0d37-44c2-96be-42dc39687913\") " pod="ai-tenant-e2e-inf-a-0bbeda/test-model-0bbeda-kserve-5cb56fbff9-lqp9j" Nov 02 01:36:22.735396 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:22.729509 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b45e955d-0d37-44c2-96be-42dc39687913-home\") pod \"test-model-0bbeda-kserve-5cb56fbff9-lqp9j\" (UID: \"b45e955d-0d37-44c2-96be-42dc39687913\") " pod="ai-tenant-e2e-inf-a-0bbeda/test-model-0bbeda-kserve-5cb56fbff9-lqp9j" Nov 02 01:36:22.735396 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:22.729630 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b45e955d-0d37-44c2-96be-42dc39687913-dshm\") pod \"test-model-0bbeda-kserve-5cb56fbff9-lqp9j\" (UID: \"b45e955d-0d37-44c2-96be-42dc39687913\") " pod="ai-tenant-e2e-inf-a-0bbeda/test-model-0bbeda-kserve-5cb56fbff9-lqp9j" Nov 02 01:36:22.735396 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:22.729738 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b45e955d-0d37-44c2-96be-42dc39687913-tmp-dir\") pod \"test-model-0bbeda-kserve-5cb56fbff9-lqp9j\" (UID: \"b45e955d-0d37-44c2-96be-42dc39687913\") " pod="ai-tenant-e2e-inf-a-0bbeda/test-model-0bbeda-kserve-5cb56fbff9-lqp9j" Nov 02 01:36:22.735396 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:22.729766 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b45e955d-0d37-44c2-96be-42dc39687913-model-cache\") pod \"test-model-0bbeda-kserve-5cb56fbff9-lqp9j\" (UID: \"b45e955d-0d37-44c2-96be-42dc39687913\") " pod="ai-tenant-e2e-inf-a-0bbeda/test-model-0bbeda-kserve-5cb56fbff9-lqp9j" Nov 02 01:36:22.735396 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:22.729830 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6bpc\" (UniqueName: \"kubernetes.io/projected/b45e955d-0d37-44c2-96be-42dc39687913-kube-api-access-k6bpc\") pod \"test-model-0bbeda-kserve-5cb56fbff9-lqp9j\" (UID: \"b45e955d-0d37-44c2-96be-42dc39687913\") " pod="ai-tenant-e2e-inf-a-0bbeda/test-model-0bbeda-kserve-5cb56fbff9-lqp9j" Nov 02 01:36:22.735396 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:22.729893 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b45e955d-0d37-44c2-96be-42dc39687913-tls-certs\") pod \"test-model-0bbeda-kserve-5cb56fbff9-lqp9j\" (UID: \"b45e955d-0d37-44c2-96be-42dc39687913\") " pod="ai-tenant-e2e-inf-a-0bbeda/test-model-0bbeda-kserve-5cb56fbff9-lqp9j" Nov 02 01:36:22.735396 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:22.730912 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b45e955d-0d37-44c2-96be-42dc39687913-tmp-dir\") pod \"test-model-0bbeda-kserve-5cb56fbff9-lqp9j\" (UID: \"b45e955d-0d37-44c2-96be-42dc39687913\") " pod="ai-tenant-e2e-inf-a-0bbeda/test-model-0bbeda-kserve-5cb56fbff9-lqp9j" Nov 02 01:36:22.735396 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:22.731158 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b45e955d-0d37-44c2-96be-42dc39687913-home\") pod \"test-model-0bbeda-kserve-5cb56fbff9-lqp9j\" (UID: \"b45e955d-0d37-44c2-96be-42dc39687913\") " pod="ai-tenant-e2e-inf-a-0bbeda/test-model-0bbeda-kserve-5cb56fbff9-lqp9j" Nov 02 01:36:22.735396 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:22.732776 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b45e955d-0d37-44c2-96be-42dc39687913-model-cache\") pod \"test-model-0bbeda-kserve-5cb56fbff9-lqp9j\" (UID: \"b45e955d-0d37-44c2-96be-42dc39687913\") " pod="ai-tenant-e2e-inf-a-0bbeda/test-model-0bbeda-kserve-5cb56fbff9-lqp9j" Nov 02 01:36:22.758451 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:22.757712 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b45e955d-0d37-44c2-96be-42dc39687913-dshm\") pod \"test-model-0bbeda-kserve-5cb56fbff9-lqp9j\" (UID: \"b45e955d-0d37-44c2-96be-42dc39687913\") " pod="ai-tenant-e2e-inf-a-0bbeda/test-model-0bbeda-kserve-5cb56fbff9-lqp9j" Nov 02 01:36:22.773185 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:22.769005 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b45e955d-0d37-44c2-96be-42dc39687913-tls-certs\") pod \"test-model-0bbeda-kserve-5cb56fbff9-lqp9j\" (UID: \"b45e955d-0d37-44c2-96be-42dc39687913\") " pod="ai-tenant-e2e-inf-a-0bbeda/test-model-0bbeda-kserve-5cb56fbff9-lqp9j" Nov 02 01:36:22.773185 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:22.769089 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6bpc\" (UniqueName: \"kubernetes.io/projected/b45e955d-0d37-44c2-96be-42dc39687913-kube-api-access-k6bpc\") pod \"test-model-0bbeda-kserve-5cb56fbff9-lqp9j\" (UID: \"b45e955d-0d37-44c2-96be-42dc39687913\") " pod="ai-tenant-e2e-inf-a-0bbeda/test-model-0bbeda-kserve-5cb56fbff9-lqp9j" Nov 02 01:36:22.845353 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:22.843761 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0bbeda/test-model-0bbeda-kserve-5cb56fbff9-lqp9j" Nov 02 01:36:23.243841 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:23.243807 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-0bbeda/test-model-0bbeda-kserve-5cb56fbff9-lqp9j"] Nov 02 01:36:23.244011 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:36:23.243975 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb45e955d_0d37_44c2_96be_42dc39687913.slice/crio-c3f4fc28cdb2e7290c6e628761636139def59a99543bead180a8fd7945d1e3cc WatchSource:0}: Error finding container c3f4fc28cdb2e7290c6e628761636139def59a99543bead180a8fd7945d1e3cc: Status 404 returned error can't find the container with id c3f4fc28cdb2e7290c6e628761636139def59a99543bead180a8fd7945d1e3cc Nov 02 01:36:23.454484 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:23.454443 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0bbeda/test-model-0bbeda-kserve-5cb56fbff9-lqp9j" event={"ID":"b45e955d-0d37-44c2-96be-42dc39687913","Type":"ContainerStarted","Data":"e34deec8bd31da8db32c1f3335f3a295677a0550b02aa961c660116bf80b680f"} Nov 02 01:36:23.454702 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:23.454498 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0bbeda/test-model-0bbeda-kserve-5cb56fbff9-lqp9j" event={"ID":"b45e955d-0d37-44c2-96be-42dc39687913","Type":"ContainerStarted","Data":"c3f4fc28cdb2e7290c6e628761636139def59a99543bead180a8fd7945d1e3cc"} Nov 02 01:36:23.475632 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:23.475581 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-0bbeda/test-model-0bbeda-kserve-5cb56fbff9-lqp9j" podStartSLOduration=1.47556739 podStartE2EDuration="1.47556739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:36:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:36:23.473573909 +0000 UTC m=+1288.094579121" watchObservedRunningTime="2026-11-02 01:36:23.47556739 +0000 UTC m=+1288.096572606" Nov 02 01:36:23.844603 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:23.844568 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-0bbeda/test-model-0bbeda-kserve-5cb56fbff9-lqp9j" Nov 02 01:36:25.559779 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:25.559740 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2589b2-5c75d66b9-fkj6b"] Nov 02 01:36:25.595464 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:25.595429 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2589b2-5c75d66b9-fkj6b" Nov 02 01:36:25.601423 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:25.601347 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2589b2-5c75d66b9-fkj6b"] Nov 02 01:36:25.602759 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:25.602732 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-2589b2-serving-cert\"" Nov 02 01:36:25.631012 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:25.630966 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3869bb0d-5d24-4a09-bd83-9aea716bfd90-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2589b2-5c75d66b9-fkj6b\" (UID: \"3869bb0d-5d24-4a09-bd83-9aea716bfd90\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2589b2-5c75d66b9-fkj6b" Nov 02 01:36:25.631279 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:25.631196 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrlxk\" (UniqueName: \"kubernetes.io/projected/3869bb0d-5d24-4a09-bd83-9aea716bfd90-kube-api-access-xrlxk\") pod \"maas-api-e2e-inf-b-2589b2-5c75d66b9-fkj6b\" (UID: \"3869bb0d-5d24-4a09-bd83-9aea716bfd90\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2589b2-5c75d66b9-fkj6b" Nov 02 01:36:25.631525 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:25.631499 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3869bb0d-5d24-4a09-bd83-9aea716bfd90-maas-api-tls\") pod \"maas-api-e2e-inf-b-2589b2-5c75d66b9-fkj6b\" (UID: \"3869bb0d-5d24-4a09-bd83-9aea716bfd90\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2589b2-5c75d66b9-fkj6b" Nov 02 01:36:25.631791 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:25.631761 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3869bb0d-5d24-4a09-bd83-9aea716bfd90-metrics-certs\") pod \"maas-api-e2e-inf-b-2589b2-5c75d66b9-fkj6b\" (UID: \"3869bb0d-5d24-4a09-bd83-9aea716bfd90\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2589b2-5c75d66b9-fkj6b" Nov 02 01:36:25.743186 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:25.742664 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3869bb0d-5d24-4a09-bd83-9aea716bfd90-maas-api-tls\") pod \"maas-api-e2e-inf-b-2589b2-5c75d66b9-fkj6b\" (UID: \"3869bb0d-5d24-4a09-bd83-9aea716bfd90\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2589b2-5c75d66b9-fkj6b" Nov 02 01:36:25.753871 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:25.753520 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3869bb0d-5d24-4a09-bd83-9aea716bfd90-metrics-certs\") pod \"maas-api-e2e-inf-b-2589b2-5c75d66b9-fkj6b\" (UID: \"3869bb0d-5d24-4a09-bd83-9aea716bfd90\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2589b2-5c75d66b9-fkj6b" Nov 02 01:36:25.753871 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:25.753831 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3869bb0d-5d24-4a09-bd83-9aea716bfd90-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2589b2-5c75d66b9-fkj6b\" (UID: \"3869bb0d-5d24-4a09-bd83-9aea716bfd90\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2589b2-5c75d66b9-fkj6b" Nov 02 01:36:25.754231 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:25.754044 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrlxk\" (UniqueName: \"kubernetes.io/projected/3869bb0d-5d24-4a09-bd83-9aea716bfd90-kube-api-access-xrlxk\") pod \"maas-api-e2e-inf-b-2589b2-5c75d66b9-fkj6b\" (UID: \"3869bb0d-5d24-4a09-bd83-9aea716bfd90\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2589b2-5c75d66b9-fkj6b" Nov 02 01:36:25.757119 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:25.757079 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3869bb0d-5d24-4a09-bd83-9aea716bfd90-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2589b2-5c75d66b9-fkj6b\" (UID: \"3869bb0d-5d24-4a09-bd83-9aea716bfd90\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2589b2-5c75d66b9-fkj6b" Nov 02 01:36:25.760267 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:25.760239 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3869bb0d-5d24-4a09-bd83-9aea716bfd90-maas-api-tls\") pod \"maas-api-e2e-inf-b-2589b2-5c75d66b9-fkj6b\" (UID: \"3869bb0d-5d24-4a09-bd83-9aea716bfd90\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2589b2-5c75d66b9-fkj6b" Nov 02 01:36:25.771447 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:25.769966 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3869bb0d-5d24-4a09-bd83-9aea716bfd90-metrics-certs\") pod \"maas-api-e2e-inf-b-2589b2-5c75d66b9-fkj6b\" (UID: \"3869bb0d-5d24-4a09-bd83-9aea716bfd90\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2589b2-5c75d66b9-fkj6b" Nov 02 01:36:25.772469 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:25.772442 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrlxk\" (UniqueName: \"kubernetes.io/projected/3869bb0d-5d24-4a09-bd83-9aea716bfd90-kube-api-access-xrlxk\") pod \"maas-api-e2e-inf-b-2589b2-5c75d66b9-fkj6b\" (UID: \"3869bb0d-5d24-4a09-bd83-9aea716bfd90\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2589b2-5c75d66b9-fkj6b" Nov 02 01:36:25.915172 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:25.915140 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2589b2-5c75d66b9-fkj6b" Nov 02 01:36:26.315639 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:26.315590 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2589b2-5c75d66b9-fkj6b"] Nov 02 01:36:26.369914 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:26.369691 4427 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 01:36:26.489272 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:26.489234 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2589b2-5c75d66b9-fkj6b" event={"ID":"3869bb0d-5d24-4a09-bd83-9aea716bfd90","Type":"ContainerStarted","Data":"094eb307744220bc21054a531a28e6577d06561c8f2410c254844d2e306e1f4f"} Nov 02 01:36:27.498663 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:27.498628 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2589b2-5c75d66b9-fkj6b" event={"ID":"3869bb0d-5d24-4a09-bd83-9aea716bfd90","Type":"ContainerStarted","Data":"e3bd476b484a40fd18b7ab169aebb10519356d01340b07ed4ce3f841ee22ef80"} Nov 02 01:36:27.499197 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:27.498765 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2589b2-5c75d66b9-fkj6b" Nov 02 01:36:27.522295 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:27.522236 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2589b2-5c75d66b9-fkj6b" podStartSLOduration=1.941359076 podStartE2EDuration="2.522222521s" totalImagesPullingTime="580.863445ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:36:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:36:27.518509013 +0000 UTC m=+1292.139514226" watchObservedRunningTime="2026-11-02 01:36:27.522222521 +0000 UTC m=+1292.143227784" Nov 02 01:36:31.586981 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:36:31.586940 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:36:32.844481 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:32.844446 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-0bbeda/test-model-0bbeda-kserve-5cb56fbff9-lqp9j" Nov 02 01:36:32.857636 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:32.857608 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-0bbeda/test-model-0bbeda-kserve-5cb56fbff9-lqp9j" Nov 02 01:36:33.515760 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:33.515725 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2589b2-5c75d66b9-fkj6b" Nov 02 01:36:33.585456 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:33.585422 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-0bbeda/test-model-0bbeda-kserve-5cb56fbff9-lqp9j" Nov 02 01:36:37.018100 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:37.018003 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7"] Nov 02 01:36:37.029771 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:37.029027 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7" Nov 02 01:36:37.035983 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:37.035926 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-9a4595-openshift-default-dockercfg-trdc7\"" Nov 02 01:36:37.062213 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:37.062132 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7"] Nov 02 01:36:37.108870 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:37.108824 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/273e0a10-0ba6-41bb-b8db-1ff672905957-credential-socket\") pod \"e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7\" (UID: \"273e0a10-0ba6-41bb-b8db-1ff672905957\") " pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7" Nov 02 01:36:37.109090 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:37.108978 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/273e0a10-0ba6-41bb-b8db-1ff672905957-istiod-ca-cert\") pod \"e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7\" (UID: \"273e0a10-0ba6-41bb-b8db-1ff672905957\") " pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7" Nov 02 01:36:37.109090 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:37.109014 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/273e0a10-0ba6-41bb-b8db-1ff672905957-istio-envoy\") pod \"e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7\" (UID: \"273e0a10-0ba6-41bb-b8db-1ff672905957\") " pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7" Nov 02 01:36:37.109090 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:37.109064 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkc7r\" (UniqueName: \"kubernetes.io/projected/273e0a10-0ba6-41bb-b8db-1ff672905957-kube-api-access-zkc7r\") pod \"e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7\" (UID: \"273e0a10-0ba6-41bb-b8db-1ff672905957\") " pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7" Nov 02 01:36:37.109264 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:37.109098 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/273e0a10-0ba6-41bb-b8db-1ff672905957-istio-podinfo\") pod \"e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7\" (UID: \"273e0a10-0ba6-41bb-b8db-1ff672905957\") " pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7" Nov 02 01:36:37.109264 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:37.109148 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/273e0a10-0ba6-41bb-b8db-1ff672905957-istio-token\") pod \"e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7\" (UID: \"273e0a10-0ba6-41bb-b8db-1ff672905957\") " pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7" Nov 02 01:36:37.109264 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:37.109225 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/273e0a10-0ba6-41bb-b8db-1ff672905957-workload-socket\") pod \"e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7\" (UID: \"273e0a10-0ba6-41bb-b8db-1ff672905957\") " pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7" Nov 02 01:36:37.109510 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:37.109350 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/273e0a10-0ba6-41bb-b8db-1ff672905957-workload-certs\") pod \"e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7\" (UID: \"273e0a10-0ba6-41bb-b8db-1ff672905957\") " pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7" Nov 02 01:36:37.109639 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:37.109619 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/273e0a10-0ba6-41bb-b8db-1ff672905957-istio-data\") pod \"e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7\" (UID: \"273e0a10-0ba6-41bb-b8db-1ff672905957\") " pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7" Nov 02 01:36:37.326454 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:37.305627 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/273e0a10-0ba6-41bb-b8db-1ff672905957-istio-data\") pod \"e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7\" (UID: \"273e0a10-0ba6-41bb-b8db-1ff672905957\") " pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7" Nov 02 01:36:37.326454 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:37.305779 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/273e0a10-0ba6-41bb-b8db-1ff672905957-credential-socket\") pod \"e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7\" (UID: \"273e0a10-0ba6-41bb-b8db-1ff672905957\") " pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7" Nov 02 01:36:37.337401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:37.328872 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/273e0a10-0ba6-41bb-b8db-1ff672905957-istio-data\") pod \"e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7\" (UID: \"273e0a10-0ba6-41bb-b8db-1ff672905957\") " pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7" Nov 02 01:36:37.337401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:37.330001 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/273e0a10-0ba6-41bb-b8db-1ff672905957-istiod-ca-cert\") pod \"e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7\" (UID: \"273e0a10-0ba6-41bb-b8db-1ff672905957\") " pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7" Nov 02 01:36:37.337401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:37.330043 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/273e0a10-0ba6-41bb-b8db-1ff672905957-istio-envoy\") pod \"e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7\" (UID: \"273e0a10-0ba6-41bb-b8db-1ff672905957\") " pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7" Nov 02 01:36:37.337401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:37.330093 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkc7r\" (UniqueName: \"kubernetes.io/projected/273e0a10-0ba6-41bb-b8db-1ff672905957-kube-api-access-zkc7r\") pod \"e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7\" (UID: \"273e0a10-0ba6-41bb-b8db-1ff672905957\") " pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7" Nov 02 01:36:37.337401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:37.330125 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/273e0a10-0ba6-41bb-b8db-1ff672905957-istio-podinfo\") pod \"e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7\" (UID: \"273e0a10-0ba6-41bb-b8db-1ff672905957\") " pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7" Nov 02 01:36:37.337401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:37.330170 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/273e0a10-0ba6-41bb-b8db-1ff672905957-istio-token\") pod \"e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7\" (UID: \"273e0a10-0ba6-41bb-b8db-1ff672905957\") " pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7" Nov 02 01:36:37.337401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:37.330242 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/273e0a10-0ba6-41bb-b8db-1ff672905957-workload-socket\") pod \"e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7\" (UID: \"273e0a10-0ba6-41bb-b8db-1ff672905957\") " pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7" Nov 02 01:36:37.337401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:37.330387 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/273e0a10-0ba6-41bb-b8db-1ff672905957-workload-certs\") pod \"e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7\" (UID: \"273e0a10-0ba6-41bb-b8db-1ff672905957\") " pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7" Nov 02 01:36:37.337401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:37.330938 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/273e0a10-0ba6-41bb-b8db-1ff672905957-workload-certs\") pod \"e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7\" (UID: \"273e0a10-0ba6-41bb-b8db-1ff672905957\") " pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7" Nov 02 01:36:37.337401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:37.331153 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/273e0a10-0ba6-41bb-b8db-1ff672905957-credential-socket\") pod \"e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7\" (UID: \"273e0a10-0ba6-41bb-b8db-1ff672905957\") " pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7" Nov 02 01:36:37.337401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:37.331770 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/273e0a10-0ba6-41bb-b8db-1ff672905957-istiod-ca-cert\") pod \"e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7\" (UID: \"273e0a10-0ba6-41bb-b8db-1ff672905957\") " pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7" Nov 02 01:36:37.340746 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:37.340404 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/273e0a10-0ba6-41bb-b8db-1ff672905957-workload-socket\") pod \"e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7\" (UID: \"273e0a10-0ba6-41bb-b8db-1ff672905957\") " pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7" Nov 02 01:36:37.422684 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:37.420870 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/273e0a10-0ba6-41bb-b8db-1ff672905957-istio-podinfo\") pod \"e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7\" (UID: \"273e0a10-0ba6-41bb-b8db-1ff672905957\") " pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7" Nov 02 01:36:37.439247 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:37.439170 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkc7r\" (UniqueName: \"kubernetes.io/projected/273e0a10-0ba6-41bb-b8db-1ff672905957-kube-api-access-zkc7r\") pod \"e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7\" (UID: \"273e0a10-0ba6-41bb-b8db-1ff672905957\") " pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7" Nov 02 01:36:37.450424 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:37.440189 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/273e0a10-0ba6-41bb-b8db-1ff672905957-istio-envoy\") pod \"e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7\" (UID: \"273e0a10-0ba6-41bb-b8db-1ff672905957\") " pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7" Nov 02 01:36:37.450424 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:37.449992 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/273e0a10-0ba6-41bb-b8db-1ff672905957-istio-token\") pod \"e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7\" (UID: \"273e0a10-0ba6-41bb-b8db-1ff672905957\") " pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7" Nov 02 01:36:37.659630 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:37.659597 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7" Nov 02 01:36:37.971989 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:37.971960 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7"] Nov 02 01:36:37.973788 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:36:37.973760 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod273e0a10_0ba6_41bb_b8db_1ff672905957.slice/crio-c8a682cf493af60a41700bd561ddd67af9f5f2a0532b3d873a9e9470a83b415b WatchSource:0}: Error finding container c8a682cf493af60a41700bd561ddd67af9f5f2a0532b3d873a9e9470a83b415b: Status 404 returned error can't find the container with id c8a682cf493af60a41700bd561ddd67af9f5f2a0532b3d873a9e9470a83b415b Nov 02 01:36:37.977802 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:37.977767 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:36:37.977890 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:37.977854 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:36:37.977945 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:37.977911 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:36:38.635589 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:38.635547 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7" event={"ID":"273e0a10-0ba6-41bb-b8db-1ff672905957","Type":"ContainerStarted","Data":"31481e93dfdddb0e8abe4b5f38d64987489513b46bdc3259dcacd8fa44b152ba"} Nov 02 01:36:38.636042 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:38.635628 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7" event={"ID":"273e0a10-0ba6-41bb-b8db-1ff672905957","Type":"ContainerStarted","Data":"c8a682cf493af60a41700bd561ddd67af9f5f2a0532b3d873a9e9470a83b415b"} Nov 02 01:36:38.660565 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:38.660525 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7" Nov 02 01:36:38.666094 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:38.666031 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7" podStartSLOduration=2.6660163150000002 podStartE2EDuration="2.666016315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:36:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:36:38.658902076 +0000 UTC m=+1303.279907288" watchObservedRunningTime="2026-11-02 01:36:38.666016315 +0000 UTC m=+1303.287021526" Nov 02 01:36:39.668483 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:39.668450 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7" Nov 02 01:36:39.781943 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:39.781904 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z"] Nov 02 01:36:39.782198 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:39.782165 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z" podUID="006dbe2b-932c-4c07-9430-51b9bd3b2bb2" containerName="istio-proxy" containerID="cri-o://dd4a4eb0c94ba307188bf1606d6451ec807a34ce04062d0daa26a941bfd1bb47" gracePeriod=30 Nov 02 01:36:39.789400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:39.787532 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97"] Nov 02 01:36:39.789400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:39.787758 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97" podUID="0c3740bf-b371-41f5-9b96-0096705bf9b9" containerName="istio-proxy" containerID="cri-o://22cac3508c764cb692391790ee4b28797982ead11e03bf6b43e6f601bc944360" gracePeriod=30 Nov 02 01:36:40.014317 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:40.014231 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp"] Nov 02 01:36:40.015531 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:40.015497 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp" podUID="1699d426-179c-4719-ac8e-1d05d2e199b8" containerName="istio-proxy" containerID="cri-o://73180bda1b3e71cfdfd749f16192b9a29a193b44ed72b24f1454fa69a757bf6b" gracePeriod=30 Nov 02 01:36:40.651745 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:40.651714 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7" Nov 02 01:36:40.653325 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:40.653305 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7" Nov 02 01:36:41.123747 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:41.123711 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-50537f4f-67d6cc6b87-7w7v4"] Nov 02 01:36:44.218524 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:44.218486 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9a4595-cc768d49f-fb6l8"] Nov 02 01:36:44.252935 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:44.252677 4427 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 01:36:44.255412 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:44.255139 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp" podUID="1699d426-179c-4719-ac8e-1d05d2e199b8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 01:36:44.664544 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:44.664503 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9a4595-cc768d49f-fb6l8"] Nov 02 01:36:44.664823 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:44.664803 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9a4595-5d57f7c66f-5vgsh"] Nov 02 01:36:44.686259 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:44.683049 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9a4595-cc768d49f-fb6l8" Nov 02 01:36:44.765236 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:44.691955 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-9a4595\"" Nov 02 01:36:44.765236 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:44.692170 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-9a4595-dockercfg-4wgf8\"" Nov 02 01:36:44.765236 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:44.714861 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9a4595-5d57f7c66f-5vgsh"] Nov 02 01:36:44.765236 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:44.714984 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9a4595-5d57f7c66f-5vgsh" Nov 02 01:36:44.868556 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:44.866328 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd3cc6d4-bb20-4c1d-a27f-b623408bafa4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9a4595-cc768d49f-fb6l8\" (UID: \"fd3cc6d4-bb20-4c1d-a27f-b623408bafa4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9a4595-cc768d49f-fb6l8" Nov 02 01:36:44.868556 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:44.867787 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j9m4\" (UniqueName: \"kubernetes.io/projected/fd3cc6d4-bb20-4c1d-a27f-b623408bafa4-kube-api-access-7j9m4\") pod \"payload-pre-processing-e2e-inf-b-9a4595-cc768d49f-fb6l8\" (UID: \"fd3cc6d4-bb20-4c1d-a27f-b623408bafa4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9a4595-cc768d49f-fb6l8" Nov 02 01:36:44.868556 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:44.867929 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53ae6505-044c-4e0d-8962-7aa6eb262ac1-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9a4595-5d57f7c66f-5vgsh\" (UID: \"53ae6505-044c-4e0d-8962-7aa6eb262ac1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9a4595-5d57f7c66f-5vgsh" Nov 02 01:36:44.868556 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:44.868016 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs9fz\" (UniqueName: \"kubernetes.io/projected/53ae6505-044c-4e0d-8962-7aa6eb262ac1-kube-api-access-xs9fz\") pod \"payload-processing-e2e-inf-b-9a4595-5d57f7c66f-5vgsh\" (UID: \"53ae6505-044c-4e0d-8962-7aa6eb262ac1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9a4595-5d57f7c66f-5vgsh" Nov 02 01:36:45.065759 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:45.058120 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd3cc6d4-bb20-4c1d-a27f-b623408bafa4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9a4595-cc768d49f-fb6l8\" (UID: \"fd3cc6d4-bb20-4c1d-a27f-b623408bafa4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9a4595-cc768d49f-fb6l8" Nov 02 01:36:45.065759 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:45.058234 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j9m4\" (UniqueName: \"kubernetes.io/projected/fd3cc6d4-bb20-4c1d-a27f-b623408bafa4-kube-api-access-7j9m4\") pod \"payload-pre-processing-e2e-inf-b-9a4595-cc768d49f-fb6l8\" (UID: \"fd3cc6d4-bb20-4c1d-a27f-b623408bafa4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9a4595-cc768d49f-fb6l8" Nov 02 01:36:45.065759 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:45.058342 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53ae6505-044c-4e0d-8962-7aa6eb262ac1-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9a4595-5d57f7c66f-5vgsh\" (UID: \"53ae6505-044c-4e0d-8962-7aa6eb262ac1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9a4595-5d57f7c66f-5vgsh" Nov 02 01:36:45.137430 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:45.134464 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs9fz\" (UniqueName: \"kubernetes.io/projected/53ae6505-044c-4e0d-8962-7aa6eb262ac1-kube-api-access-xs9fz\") pod \"payload-processing-e2e-inf-b-9a4595-5d57f7c66f-5vgsh\" (UID: \"53ae6505-044c-4e0d-8962-7aa6eb262ac1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9a4595-5d57f7c66f-5vgsh" Nov 02 01:36:45.137430 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:45.136243 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd3cc6d4-bb20-4c1d-a27f-b623408bafa4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9a4595-cc768d49f-fb6l8\" (UID: \"fd3cc6d4-bb20-4c1d-a27f-b623408bafa4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9a4595-cc768d49f-fb6l8" Nov 02 01:36:45.137430 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:45.137352 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53ae6505-044c-4e0d-8962-7aa6eb262ac1-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9a4595-5d57f7c66f-5vgsh\" (UID: \"53ae6505-044c-4e0d-8962-7aa6eb262ac1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9a4595-5d57f7c66f-5vgsh" Nov 02 01:36:45.412520 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:45.378776 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" podUID="f67c954a-ce5b-41d9-8d99-7d126bd89d59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:36:45.459428 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:45.456320 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs9fz\" (UniqueName: \"kubernetes.io/projected/53ae6505-044c-4e0d-8962-7aa6eb262ac1-kube-api-access-xs9fz\") pod \"payload-processing-e2e-inf-b-9a4595-5d57f7c66f-5vgsh\" (UID: \"53ae6505-044c-4e0d-8962-7aa6eb262ac1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9a4595-5d57f7c66f-5vgsh" Nov 02 01:36:45.544026 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:45.540114 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j9m4\" (UniqueName: \"kubernetes.io/projected/fd3cc6d4-bb20-4c1d-a27f-b623408bafa4-kube-api-access-7j9m4\") pod \"payload-pre-processing-e2e-inf-b-9a4595-cc768d49f-fb6l8\" (UID: \"fd3cc6d4-bb20-4c1d-a27f-b623408bafa4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9a4595-cc768d49f-fb6l8" Nov 02 01:36:45.628065 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:45.627749 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9a4595-cc768d49f-fb6l8" Nov 02 01:36:45.653780 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:45.646221 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" podUID="cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:36:45.717626 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:45.699240 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" podUID="353a8a41-21ee-4470-861c-b53833beb106" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 02 01:36:45.717626 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:45.693342 4427 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" podUID="366c78ee-4aaf-4b7c-84b4-17028538cb1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:36:45.717626 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:45.716859 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" podUID="9c056798-2cf8-47b1-9829-67cebda7a248" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 01:36:45.750420 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:45.740616 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9a4595-5d57f7c66f-5vgsh" Nov 02 01:36:45.750420 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:45.749724 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" podUID="706df1bd-19cf-4ab9-9aaa-3c12c97ccf45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 01:36:46.126603 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:46.125878 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-50537f4f-67d6cc6b87-7w7v4" podUID="6e387f69-ed5c-449a-9ba5-8e03a04b235a" containerName="payload-processing" containerID="cri-o://8cd30d6081c67e1aaed4ca03165ed4b107615abcc5842066617aa70c153bc0b0" gracePeriod=25 Nov 02 01:36:46.373145 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:46.371515 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97" Nov 02 01:36:46.590069 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:46.585335 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c3740bf-b371-41f5-9b96-0096705bf9b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c3740bf-b371-41f5-9b96-0096705bf9b9-istio-data\") pod \"0c3740bf-b371-41f5-9b96-0096705bf9b9\" (UID: \"0c3740bf-b371-41f5-9b96-0096705bf9b9\") " Nov 02 01:36:46.590069 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:46.585413 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c3740bf-b371-41f5-9b96-0096705bf9b9-kube-api-access-f7nhm\" (UniqueName: \"kubernetes.io/projected/0c3740bf-b371-41f5-9b96-0096705bf9b9-kube-api-access-f7nhm\") pod \"0c3740bf-b371-41f5-9b96-0096705bf9b9\" (UID: \"0c3740bf-b371-41f5-9b96-0096705bf9b9\") " Nov 02 01:36:46.590069 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:46.585544 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c3740bf-b371-41f5-9b96-0096705bf9b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c3740bf-b371-41f5-9b96-0096705bf9b9-istiod-ca-cert\") pod \"0c3740bf-b371-41f5-9b96-0096705bf9b9\" (UID: \"0c3740bf-b371-41f5-9b96-0096705bf9b9\") " Nov 02 01:36:46.590069 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:46.585570 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c3740bf-b371-41f5-9b96-0096705bf9b9-istio-token\" (UniqueName: \"kubernetes.io/projected/0c3740bf-b371-41f5-9b96-0096705bf9b9-istio-token\") pod \"0c3740bf-b371-41f5-9b96-0096705bf9b9\" (UID: \"0c3740bf-b371-41f5-9b96-0096705bf9b9\") " Nov 02 01:36:46.590069 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:46.585639 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c3740bf-b371-41f5-9b96-0096705bf9b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c3740bf-b371-41f5-9b96-0096705bf9b9-istio-podinfo\") pod \"0c3740bf-b371-41f5-9b96-0096705bf9b9\" (UID: \"0c3740bf-b371-41f5-9b96-0096705bf9b9\") " Nov 02 01:36:46.590069 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:46.585666 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c3740bf-b371-41f5-9b96-0096705bf9b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c3740bf-b371-41f5-9b96-0096705bf9b9-workload-certs\") pod \"0c3740bf-b371-41f5-9b96-0096705bf9b9\" (UID: \"0c3740bf-b371-41f5-9b96-0096705bf9b9\") " Nov 02 01:36:46.590069 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:46.585815 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c3740bf-b371-41f5-9b96-0096705bf9b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c3740bf-b371-41f5-9b96-0096705bf9b9-istio-envoy\") pod \"0c3740bf-b371-41f5-9b96-0096705bf9b9\" (UID: \"0c3740bf-b371-41f5-9b96-0096705bf9b9\") " Nov 02 01:36:46.590069 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:46.585853 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c3740bf-b371-41f5-9b96-0096705bf9b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c3740bf-b371-41f5-9b96-0096705bf9b9-credential-socket\") pod \"0c3740bf-b371-41f5-9b96-0096705bf9b9\" (UID: \"0c3740bf-b371-41f5-9b96-0096705bf9b9\") " Nov 02 01:36:46.590069 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:46.585957 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c3740bf-b371-41f5-9b96-0096705bf9b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c3740bf-b371-41f5-9b96-0096705bf9b9-workload-socket\") pod \"0c3740bf-b371-41f5-9b96-0096705bf9b9\" (UID: \"0c3740bf-b371-41f5-9b96-0096705bf9b9\") " Nov 02 01:36:46.601983 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:46.600679 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c3740bf-b371-41f5-9b96-0096705bf9b9-workload-socket" pod "0c3740bf-b371-41f5-9b96-0096705bf9b9" (UID: "0c3740bf-b371-41f5-9b96-0096705bf9b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:36:46.602631 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:46.602540 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c3740bf-b371-41f5-9b96-0096705bf9b9-credential-socket" pod "0c3740bf-b371-41f5-9b96-0096705bf9b9" (UID: "0c3740bf-b371-41f5-9b96-0096705bf9b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:36:46.607393 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:46.605731 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c3740bf-b371-41f5-9b96-0096705bf9b9-istiod-ca-cert" pod "0c3740bf-b371-41f5-9b96-0096705bf9b9" (UID: "0c3740bf-b371-41f5-9b96-0096705bf9b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:36:46.607393 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:46.606015 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c3740bf-b371-41f5-9b96-0096705bf9b9-istio-data" pod "0c3740bf-b371-41f5-9b96-0096705bf9b9" (UID: "0c3740bf-b371-41f5-9b96-0096705bf9b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:36:46.607704 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:46.607653 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c3740bf-b371-41f5-9b96-0096705bf9b9-workload-certs" pod "0c3740bf-b371-41f5-9b96-0096705bf9b9" (UID: "0c3740bf-b371-41f5-9b96-0096705bf9b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:36:46.653311 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:46.653114 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" podUID="cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:36:46.653879 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:46.653846 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" Nov 02 01:36:46.695754 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:46.684295 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c3740bf-b371-41f5-9b96-0096705bf9b9-istio-envoy" pod "0c3740bf-b371-41f5-9b96-0096705bf9b9" (UID: "0c3740bf-b371-41f5-9b96-0096705bf9b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:36:46.715867 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:46.701549 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c3740bf-b371-41f5-9b96-0096705bf9b9-kube-api-access-f7nhm" pod "0c3740bf-b371-41f5-9b96-0096705bf9b9" (UID: "0c3740bf-b371-41f5-9b96-0096705bf9b9"). InnerVolumeSpecName "kube-api-access-f7nhm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:36:46.747425 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:46.736073 4427 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c3740bf-b371-41f5-9b96-0096705bf9b9-workload-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:46.807529 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:46.807457 4427 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c3740bf-b371-41f5-9b96-0096705bf9b9-istio-data\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:46.807755 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:46.807738 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7nhm\" (UniqueName: \"kubernetes.io/projected/0c3740bf-b371-41f5-9b96-0096705bf9b9-kube-api-access-f7nhm\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:46.825934 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:46.807858 4427 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c3740bf-b371-41f5-9b96-0096705bf9b9-istiod-ca-cert\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:46.826447 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:46.826423 4427 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c3740bf-b371-41f5-9b96-0096705bf9b9-workload-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:46.826590 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:46.826576 4427 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c3740bf-b371-41f5-9b96-0096705bf9b9-istio-envoy\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:46.844551 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:46.809786 4427 generic.go:356] "Generic (PLEG): container finished" podID="1699d426-179c-4719-ac8e-1d05d2e199b8" containerID="73180bda1b3e71cfdfd749f16192b9a29a193b44ed72b24f1454fa69a757bf6b" exitCode=0 Nov 02 01:36:46.844551 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:46.806073 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c3740bf-b371-41f5-9b96-0096705bf9b9-istio-podinfo" pod "0c3740bf-b371-41f5-9b96-0096705bf9b9" (UID: "0c3740bf-b371-41f5-9b96-0096705bf9b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:36:46.844551 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:46.834725 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z" Nov 02 01:36:46.844551 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:46.834521 4427 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c3740bf-b371-41f5-9b96-0096705bf9b9-credential-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:46.844551 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:46.814645 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c3740bf-b371-41f5-9b96-0096705bf9b9-istio-token" pod "0c3740bf-b371-41f5-9b96-0096705bf9b9" (UID: "0c3740bf-b371-41f5-9b96-0096705bf9b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:36:46.844551 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:46.756427 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" Nov 02 01:36:46.844551 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:46.835160 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp" event={"ID":"1699d426-179c-4719-ac8e-1d05d2e199b8","Type":"ContainerDied","Data":"73180bda1b3e71cfdfd749f16192b9a29a193b44ed72b24f1454fa69a757bf6b"} Nov 02 01:36:46.844551 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:46.837659 4427 generic.go:356] "Generic (PLEG): container finished" podID="006dbe2b-932c-4c07-9430-51b9bd3b2bb2" containerID="dd4a4eb0c94ba307188bf1606d6451ec807a34ce04062d0daa26a941bfd1bb47" exitCode=0 Nov 02 01:36:46.844551 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:46.837805 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z" event={"ID":"006dbe2b-932c-4c07-9430-51b9bd3b2bb2","Type":"ContainerDied","Data":"dd4a4eb0c94ba307188bf1606d6451ec807a34ce04062d0daa26a941bfd1bb47"} Nov 02 01:36:46.844551 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:46.837859 4427 scope.go:122] "RemoveContainer" containerID="dd4a4eb0c94ba307188bf1606d6451ec807a34ce04062d0daa26a941bfd1bb47" Nov 02 01:36:46.850253 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:46.845892 4427 generic.go:356] "Generic (PLEG): container finished" podID="0c3740bf-b371-41f5-9b96-0096705bf9b9" containerID="22cac3508c764cb692391790ee4b28797982ead11e03bf6b43e6f601bc944360" exitCode=0 Nov 02 01:36:46.850253 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:46.845945 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97" event={"ID":"0c3740bf-b371-41f5-9b96-0096705bf9b9","Type":"ContainerDied","Data":"22cac3508c764cb692391790ee4b28797982ead11e03bf6b43e6f601bc944360"} Nov 02 01:36:46.850253 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:46.845988 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97" event={"ID":"0c3740bf-b371-41f5-9b96-0096705bf9b9","Type":"ContainerDied","Data":"b183009234955ccb319309de9c95edd3296a08e98357218671f724d38bf60fa3"} Nov 02 01:36:46.850253 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:46.846069 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97" Nov 02 01:36:46.948494 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:46.944948 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/006dbe2b-932c-4c07-9430-51b9bd3b2bb2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/006dbe2b-932c-4c07-9430-51b9bd3b2bb2-istio-envoy\") pod \"006dbe2b-932c-4c07-9430-51b9bd3b2bb2\" (UID: \"006dbe2b-932c-4c07-9430-51b9bd3b2bb2\") " Nov 02 01:36:46.948494 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:46.945040 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/006dbe2b-932c-4c07-9430-51b9bd3b2bb2-kube-api-access-92d8g\" (UniqueName: \"kubernetes.io/projected/006dbe2b-932c-4c07-9430-51b9bd3b2bb2-kube-api-access-92d8g\") pod \"006dbe2b-932c-4c07-9430-51b9bd3b2bb2\" (UID: \"006dbe2b-932c-4c07-9430-51b9bd3b2bb2\") " Nov 02 01:36:46.948494 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:46.945088 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/006dbe2b-932c-4c07-9430-51b9bd3b2bb2-istio-token\" (UniqueName: \"kubernetes.io/projected/006dbe2b-932c-4c07-9430-51b9bd3b2bb2-istio-token\") pod \"006dbe2b-932c-4c07-9430-51b9bd3b2bb2\" (UID: \"006dbe2b-932c-4c07-9430-51b9bd3b2bb2\") " Nov 02 01:36:46.948494 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:46.945136 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/006dbe2b-932c-4c07-9430-51b9bd3b2bb2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/006dbe2b-932c-4c07-9430-51b9bd3b2bb2-istio-data\") pod \"006dbe2b-932c-4c07-9430-51b9bd3b2bb2\" (UID: \"006dbe2b-932c-4c07-9430-51b9bd3b2bb2\") " Nov 02 01:36:46.948494 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:46.945185 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/006dbe2b-932c-4c07-9430-51b9bd3b2bb2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/006dbe2b-932c-4c07-9430-51b9bd3b2bb2-istio-podinfo\") pod \"006dbe2b-932c-4c07-9430-51b9bd3b2bb2\" (UID: \"006dbe2b-932c-4c07-9430-51b9bd3b2bb2\") " Nov 02 01:36:46.948494 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:46.945276 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/006dbe2b-932c-4c07-9430-51b9bd3b2bb2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/006dbe2b-932c-4c07-9430-51b9bd3b2bb2-workload-certs\") pod \"006dbe2b-932c-4c07-9430-51b9bd3b2bb2\" (UID: \"006dbe2b-932c-4c07-9430-51b9bd3b2bb2\") " Nov 02 01:36:46.948494 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:46.945515 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/006dbe2b-932c-4c07-9430-51b9bd3b2bb2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/006dbe2b-932c-4c07-9430-51b9bd3b2bb2-credential-socket\") pod \"006dbe2b-932c-4c07-9430-51b9bd3b2bb2\" (UID: \"006dbe2b-932c-4c07-9430-51b9bd3b2bb2\") " Nov 02 01:36:46.948494 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:46.945641 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/006dbe2b-932c-4c07-9430-51b9bd3b2bb2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/006dbe2b-932c-4c07-9430-51b9bd3b2bb2-istiod-ca-cert\") pod \"006dbe2b-932c-4c07-9430-51b9bd3b2bb2\" (UID: \"006dbe2b-932c-4c07-9430-51b9bd3b2bb2\") " Nov 02 01:36:46.948494 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:46.945718 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/006dbe2b-932c-4c07-9430-51b9bd3b2bb2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/006dbe2b-932c-4c07-9430-51b9bd3b2bb2-workload-socket\") pod \"006dbe2b-932c-4c07-9430-51b9bd3b2bb2\" (UID: \"006dbe2b-932c-4c07-9430-51b9bd3b2bb2\") " Nov 02 01:36:46.948494 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:46.946631 4427 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c3740bf-b371-41f5-9b96-0096705bf9b9-istio-token\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:46.948494 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:46.946654 4427 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c3740bf-b371-41f5-9b96-0096705bf9b9-istio-podinfo\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:46.958628 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:46.956792 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/006dbe2b-932c-4c07-9430-51b9bd3b2bb2-workload-certs" pod "006dbe2b-932c-4c07-9430-51b9bd3b2bb2" (UID: "006dbe2b-932c-4c07-9430-51b9bd3b2bb2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:36:46.958628 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:46.957716 4427 scope.go:122] "RemoveContainer" containerID="22cac3508c764cb692391790ee4b28797982ead11e03bf6b43e6f601bc944360" Nov 02 01:36:46.964297 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:46.964039 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/006dbe2b-932c-4c07-9430-51b9bd3b2bb2-istiod-ca-cert" pod "006dbe2b-932c-4c07-9430-51b9bd3b2bb2" (UID: "006dbe2b-932c-4c07-9430-51b9bd3b2bb2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:36:46.991365 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:46.964512 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/006dbe2b-932c-4c07-9430-51b9bd3b2bb2-credential-socket" pod "006dbe2b-932c-4c07-9430-51b9bd3b2bb2" (UID: "006dbe2b-932c-4c07-9430-51b9bd3b2bb2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:36:46.991365 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:46.966510 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/006dbe2b-932c-4c07-9430-51b9bd3b2bb2-istio-data" pod "006dbe2b-932c-4c07-9430-51b9bd3b2bb2" (UID: "006dbe2b-932c-4c07-9430-51b9bd3b2bb2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:36:46.991365 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:46.976920 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/006dbe2b-932c-4c07-9430-51b9bd3b2bb2-istio-podinfo" pod "006dbe2b-932c-4c07-9430-51b9bd3b2bb2" (UID: "006dbe2b-932c-4c07-9430-51b9bd3b2bb2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:36:47.072762 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:47.051324 4427 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/006dbe2b-932c-4c07-9430-51b9bd3b2bb2-istio-data\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:47.072762 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:47.051364 4427 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/006dbe2b-932c-4c07-9430-51b9bd3b2bb2-istio-podinfo\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:47.072762 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:47.051402 4427 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/006dbe2b-932c-4c07-9430-51b9bd3b2bb2-workload-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:47.072762 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:47.051416 4427 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/006dbe2b-932c-4c07-9430-51b9bd3b2bb2-credential-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:47.072762 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:47.051430 4427 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/006dbe2b-932c-4c07-9430-51b9bd3b2bb2-istiod-ca-cert\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:47.095839 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:47.095783 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/006dbe2b-932c-4c07-9430-51b9bd3b2bb2-kube-api-access-92d8g" pod "006dbe2b-932c-4c07-9430-51b9bd3b2bb2" (UID: "006dbe2b-932c-4c07-9430-51b9bd3b2bb2"). InnerVolumeSpecName "kube-api-access-92d8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:36:47.102208 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:47.101598 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/006dbe2b-932c-4c07-9430-51b9bd3b2bb2-workload-socket" pod "006dbe2b-932c-4c07-9430-51b9bd3b2bb2" (UID: "006dbe2b-932c-4c07-9430-51b9bd3b2bb2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:36:47.157445 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:47.157355 4427 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/006dbe2b-932c-4c07-9430-51b9bd3b2bb2-workload-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:47.157663 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:47.157647 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92d8g\" (UniqueName: \"kubernetes.io/projected/006dbe2b-932c-4c07-9430-51b9bd3b2bb2-kube-api-access-92d8g\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:47.229598 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:47.229570 4427 scope.go:122] "RemoveContainer" containerID="22cac3508c764cb692391790ee4b28797982ead11e03bf6b43e6f601bc944360" Nov 02 01:36:47.232358 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:36:47.232316 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22cac3508c764cb692391790ee4b28797982ead11e03bf6b43e6f601bc944360\": container with ID starting with 22cac3508c764cb692391790ee4b28797982ead11e03bf6b43e6f601bc944360 not found: ID does not exist" containerID="22cac3508c764cb692391790ee4b28797982ead11e03bf6b43e6f601bc944360" Nov 02 01:36:47.232633 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:47.232604 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22cac3508c764cb692391790ee4b28797982ead11e03bf6b43e6f601bc944360"} err="failed to get container status \"22cac3508c764cb692391790ee4b28797982ead11e03bf6b43e6f601bc944360\": rpc error: code = NotFound desc = could not find container \"22cac3508c764cb692391790ee4b28797982ead11e03bf6b43e6f601bc944360\": container with ID starting with 22cac3508c764cb692391790ee4b28797982ead11e03bf6b43e6f601bc944360 not found: ID does not exist" Nov 02 01:36:47.348427 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:47.348286 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" podUID="f67c954a-ce5b-41d9-8d99-7d126bd89d59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:36:47.350748 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:36:47.350704 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd3cc6d4_bb20_4c1d_a27f_b623408bafa4.slice/crio-de77d2a86c1970e6a33983c012564b2c8d7a2c26e5a0282a2f08fb87e01abacf WatchSource:0}: Error finding container de77d2a86c1970e6a33983c012564b2c8d7a2c26e5a0282a2f08fb87e01abacf: Status 404 returned error can't find the container with id de77d2a86c1970e6a33983c012564b2c8d7a2c26e5a0282a2f08fb87e01abacf Nov 02 01:36:47.538910 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:47.538851 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/006dbe2b-932c-4c07-9430-51b9bd3b2bb2-istio-envoy" pod "006dbe2b-932c-4c07-9430-51b9bd3b2bb2" (UID: "006dbe2b-932c-4c07-9430-51b9bd3b2bb2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:36:47.559001 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:47.558935 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/006dbe2b-932c-4c07-9430-51b9bd3b2bb2-istio-token" pod "006dbe2b-932c-4c07-9430-51b9bd3b2bb2" (UID: "006dbe2b-932c-4c07-9430-51b9bd3b2bb2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:36:47.600010 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:47.579358 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp" Nov 02 01:36:47.646333 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:47.639257 4427 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/006dbe2b-932c-4c07-9430-51b9bd3b2bb2-istio-envoy\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:47.646333 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:47.639295 4427 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/006dbe2b-932c-4c07-9430-51b9bd3b2bb2-istio-token\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:47.925051 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:47.898457 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1699d426-179c-4719-ac8e-1d05d2e199b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1699d426-179c-4719-ac8e-1d05d2e199b8-istiod-ca-cert\") pod \"1699d426-179c-4719-ac8e-1d05d2e199b8\" (UID: \"1699d426-179c-4719-ac8e-1d05d2e199b8\") " Nov 02 01:36:47.925051 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:47.898653 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1699d426-179c-4719-ac8e-1d05d2e199b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1699d426-179c-4719-ac8e-1d05d2e199b8-workload-certs\") pod \"1699d426-179c-4719-ac8e-1d05d2e199b8\" (UID: \"1699d426-179c-4719-ac8e-1d05d2e199b8\") " Nov 02 01:36:47.925051 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:47.898748 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1699d426-179c-4719-ac8e-1d05d2e199b8-kube-api-access-gsc2b\" (UniqueName: \"kubernetes.io/projected/1699d426-179c-4719-ac8e-1d05d2e199b8-kube-api-access-gsc2b\") pod \"1699d426-179c-4719-ac8e-1d05d2e199b8\" (UID: \"1699d426-179c-4719-ac8e-1d05d2e199b8\") " Nov 02 01:36:47.925051 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:47.898791 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1699d426-179c-4719-ac8e-1d05d2e199b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1699d426-179c-4719-ac8e-1d05d2e199b8-istio-envoy\") pod \"1699d426-179c-4719-ac8e-1d05d2e199b8\" (UID: \"1699d426-179c-4719-ac8e-1d05d2e199b8\") " Nov 02 01:36:47.925051 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:47.898852 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1699d426-179c-4719-ac8e-1d05d2e199b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1699d426-179c-4719-ac8e-1d05d2e199b8-credential-socket\") pod \"1699d426-179c-4719-ac8e-1d05d2e199b8\" (UID: \"1699d426-179c-4719-ac8e-1d05d2e199b8\") " Nov 02 01:36:47.925051 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:47.898919 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1699d426-179c-4719-ac8e-1d05d2e199b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1699d426-179c-4719-ac8e-1d05d2e199b8-istio-podinfo\") pod \"1699d426-179c-4719-ac8e-1d05d2e199b8\" (UID: \"1699d426-179c-4719-ac8e-1d05d2e199b8\") " Nov 02 01:36:47.925051 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:47.898985 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1699d426-179c-4719-ac8e-1d05d2e199b8-istio-token\" (UniqueName: \"kubernetes.io/projected/1699d426-179c-4719-ac8e-1d05d2e199b8-istio-token\") pod \"1699d426-179c-4719-ac8e-1d05d2e199b8\" (UID: \"1699d426-179c-4719-ac8e-1d05d2e199b8\") " Nov 02 01:36:47.925051 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:47.899099 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1699d426-179c-4719-ac8e-1d05d2e199b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1699d426-179c-4719-ac8e-1d05d2e199b8-workload-socket\") pod \"1699d426-179c-4719-ac8e-1d05d2e199b8\" (UID: \"1699d426-179c-4719-ac8e-1d05d2e199b8\") " Nov 02 01:36:47.925051 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:47.899222 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1699d426-179c-4719-ac8e-1d05d2e199b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1699d426-179c-4719-ac8e-1d05d2e199b8-istio-data\") pod \"1699d426-179c-4719-ac8e-1d05d2e199b8\" (UID: \"1699d426-179c-4719-ac8e-1d05d2e199b8\") " Nov 02 01:36:47.925051 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:47.913341 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1699d426-179c-4719-ac8e-1d05d2e199b8-istio-data" pod "1699d426-179c-4719-ac8e-1d05d2e199b8" (UID: "1699d426-179c-4719-ac8e-1d05d2e199b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:36:47.925051 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:47.914572 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1699d426-179c-4719-ac8e-1d05d2e199b8-credential-socket" pod "1699d426-179c-4719-ac8e-1d05d2e199b8" (UID: "1699d426-179c-4719-ac8e-1d05d2e199b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:36:47.947958 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:47.947916 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1699d426-179c-4719-ac8e-1d05d2e199b8-istiod-ca-cert" pod "1699d426-179c-4719-ac8e-1d05d2e199b8" (UID: "1699d426-179c-4719-ac8e-1d05d2e199b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:36:47.948549 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:47.948509 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1699d426-179c-4719-ac8e-1d05d2e199b8-workload-certs" pod "1699d426-179c-4719-ac8e-1d05d2e199b8" (UID: "1699d426-179c-4719-ac8e-1d05d2e199b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:36:47.961554 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:47.951605 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1699d426-179c-4719-ac8e-1d05d2e199b8-kube-api-access-gsc2b" pod "1699d426-179c-4719-ac8e-1d05d2e199b8" (UID: "1699d426-179c-4719-ac8e-1d05d2e199b8"). InnerVolumeSpecName "kube-api-access-gsc2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:36:48.054977 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:48.054943 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9a4595-cc768d49f-fb6l8"] Nov 02 01:36:48.055272 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:48.055257 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97"] Nov 02 01:36:48.072974 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:48.072940 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-t9h97"] Nov 02 01:36:48.108423 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:48.086278 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1699d426-179c-4719-ac8e-1d05d2e199b8-istio-envoy" pod "1699d426-179c-4719-ac8e-1d05d2e199b8" (UID: "1699d426-179c-4719-ac8e-1d05d2e199b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:36:48.108423 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:48.091172 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1699d426-179c-4719-ac8e-1d05d2e199b8-istio-token" pod "1699d426-179c-4719-ac8e-1d05d2e199b8" (UID: "1699d426-179c-4719-ac8e-1d05d2e199b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:36:48.108423 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:48.091862 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1699d426-179c-4719-ac8e-1d05d2e199b8-istio-podinfo" pod "1699d426-179c-4719-ac8e-1d05d2e199b8" (UID: "1699d426-179c-4719-ac8e-1d05d2e199b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:36:48.108423 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:48.092181 4427 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1699d426-179c-4719-ac8e-1d05d2e199b8-workload-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:48.108423 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:48.092202 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsc2b\" (UniqueName: \"kubernetes.io/projected/1699d426-179c-4719-ac8e-1d05d2e199b8-kube-api-access-gsc2b\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:48.108423 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:48.092218 4427 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1699d426-179c-4719-ac8e-1d05d2e199b8-credential-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:48.108423 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:48.092233 4427 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1699d426-179c-4719-ac8e-1d05d2e199b8-istio-data\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:48.108423 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:48.092271 4427 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1699d426-179c-4719-ac8e-1d05d2e199b8-istiod-ca-cert\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:48.165226 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:48.165115 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1699d426-179c-4719-ac8e-1d05d2e199b8-workload-socket" pod "1699d426-179c-4719-ac8e-1d05d2e199b8" (UID: "1699d426-179c-4719-ac8e-1d05d2e199b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:36:48.302414 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:48.294998 4427 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1699d426-179c-4719-ac8e-1d05d2e199b8-istio-envoy\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:48.302414 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:48.295031 4427 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1699d426-179c-4719-ac8e-1d05d2e199b8-istio-podinfo\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:48.302414 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:48.295058 4427 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1699d426-179c-4719-ac8e-1d05d2e199b8-istio-token\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:48.302414 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:48.295072 4427 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1699d426-179c-4719-ac8e-1d05d2e199b8-workload-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:48.311490 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:48.311448 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9a4595-5d57f7c66f-5vgsh"] Nov 02 01:36:48.348044 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:48.347672 4427 generic.go:356] "Generic (PLEG): container finished" podID="6e387f69-ed5c-449a-9ba5-8e03a04b235a" containerID="8cd30d6081c67e1aaed4ca03165ed4b107615abcc5842066617aa70c153bc0b0" exitCode=0 Nov 02 01:36:48.348044 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:48.347812 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-50537f4f-67d6cc6b87-7w7v4" event={"ID":"6e387f69-ed5c-449a-9ba5-8e03a04b235a","Type":"ContainerDied","Data":"8cd30d6081c67e1aaed4ca03165ed4b107615abcc5842066617aa70c153bc0b0"} Nov 02 01:36:48.353511 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:48.353437 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9a4595-cc768d49f-fb6l8" event={"ID":"fd3cc6d4-bb20-4c1d-a27f-b623408bafa4","Type":"ContainerStarted","Data":"de77d2a86c1970e6a33983c012564b2c8d7a2c26e5a0282a2f08fb87e01abacf"} Nov 02 01:36:48.408495 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:48.404604 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp" event={"ID":"1699d426-179c-4719-ac8e-1d05d2e199b8","Type":"ContainerDied","Data":"cfb3378c880b4dbf8ac1fd5f328ec079a5ffd5e7d0c53f5abadec622447d79c5"} Nov 02 01:36:48.408495 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:48.404715 4427 scope.go:122] "RemoveContainer" containerID="73180bda1b3e71cfdfd749f16192b9a29a193b44ed72b24f1454fa69a757bf6b" Nov 02 01:36:48.408495 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:48.404907 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp" Nov 02 01:36:48.452738 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:48.434250 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z" event={"ID":"006dbe2b-932c-4c07-9430-51b9bd3b2bb2","Type":"ContainerDied","Data":"6f852bde404675ef972aeae79f52bdc4e78ff731e8f2a38ddfc377ca5929e782"} Nov 02 01:36:48.452738 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:48.434450 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z" Nov 02 01:36:48.659418 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:48.658707 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z"] Nov 02 01:36:48.688812 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:48.664642 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-fzx9z"] Nov 02 01:36:48.738408 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:48.737945 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp"] Nov 02 01:36:48.738408 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:48.738007 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-wqtnp"] Nov 02 01:36:48.826596 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:48.826567 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-50537f4f-67d6cc6b87-7w7v4" Nov 02 01:36:48.940033 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:48.930054 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e387f69-ed5c-449a-9ba5-8e03a04b235a-kube-api-access-l7znw\" (UniqueName: \"kubernetes.io/projected/6e387f69-ed5c-449a-9ba5-8e03a04b235a-kube-api-access-l7znw\") pod \"6e387f69-ed5c-449a-9ba5-8e03a04b235a\" (UID: \"6e387f69-ed5c-449a-9ba5-8e03a04b235a\") " Nov 02 01:36:48.940033 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:48.930112 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e387f69-ed5c-449a-9ba5-8e03a04b235a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e387f69-ed5c-449a-9ba5-8e03a04b235a-plugins-config-volume\") pod \"6e387f69-ed5c-449a-9ba5-8e03a04b235a\" (UID: \"6e387f69-ed5c-449a-9ba5-8e03a04b235a\") " Nov 02 01:36:48.940033 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:48.933972 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e387f69-ed5c-449a-9ba5-8e03a04b235a-plugins-config-volume" pod "6e387f69-ed5c-449a-9ba5-8e03a04b235a" (UID: "6e387f69-ed5c-449a-9ba5-8e03a04b235a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:36:48.940033 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:48.939612 4427 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e387f69-ed5c-449a-9ba5-8e03a04b235a-plugins-config-volume\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:48.944988 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:36:48.944956 4427 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod006dbe2b_932c_4c07_9430_51b9bd3b2bb2.slice\": RecentStats: unable to find data in memory cache]" Nov 02 01:36:48.945262 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:48.945121 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e387f69-ed5c-449a-9ba5-8e03a04b235a-kube-api-access-l7znw" pod "6e387f69-ed5c-449a-9ba5-8e03a04b235a" (UID: "6e387f69-ed5c-449a-9ba5-8e03a04b235a"). InnerVolumeSpecName "kube-api-access-l7znw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:36:49.041562 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:49.041473 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7znw\" (UniqueName: \"kubernetes.io/projected/6e387f69-ed5c-449a-9ba5-8e03a04b235a-kube-api-access-l7znw\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:36:49.472206 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:49.471007 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9a4595-cc768d49f-fb6l8" event={"ID":"fd3cc6d4-bb20-4c1d-a27f-b623408bafa4","Type":"ContainerStarted","Data":"d57cc647a5acf0d4dc1fdbfaf39d66e0cfc2d82b3a7982df3296a3cec576e614"} Nov 02 01:36:49.472417 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:49.472298 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9a4595-cc768d49f-fb6l8" Nov 02 01:36:49.493041 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:49.483843 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-50537f4f-67d6cc6b87-7w7v4" event={"ID":"6e387f69-ed5c-449a-9ba5-8e03a04b235a","Type":"ContainerDied","Data":"4d2b373f50d019271bfe8ef5797c7619c5ffa5885c188f01cb9a296a76ac24f9"} Nov 02 01:36:49.493041 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:49.483915 4427 scope.go:122] "RemoveContainer" containerID="8cd30d6081c67e1aaed4ca03165ed4b107615abcc5842066617aa70c153bc0b0" Nov 02 01:36:49.493041 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:49.484031 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-50537f4f-67d6cc6b87-7w7v4" Nov 02 01:36:49.507398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:49.498240 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9a4595-5d57f7c66f-5vgsh" event={"ID":"53ae6505-044c-4e0d-8962-7aa6eb262ac1","Type":"ContainerStarted","Data":"c6e7df11e1809ac34f6e438c3e7925fa5b8f9b2b74353a2fe2a8ffdf9c80b0b6"} Nov 02 01:36:49.507398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:49.498311 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9a4595-5d57f7c66f-5vgsh" event={"ID":"53ae6505-044c-4e0d-8962-7aa6eb262ac1","Type":"ContainerStarted","Data":"891cef75aca2ce081b49aa940f042a1783d60a2318576ff41b16e4dbb2175c9b"} Nov 02 01:36:49.507398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:49.499312 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-9a4595-5d57f7c66f-5vgsh" Nov 02 01:36:49.529029 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:49.528983 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9a4595-cc768d49f-fb6l8" podStartSLOduration=5.52896892 podStartE2EDuration="5.52896892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:36:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:36:49.526729842 +0000 UTC m=+1314.147735076" watchObservedRunningTime="2026-11-02 01:36:49.52896892 +0000 UTC m=+1314.149974150" Nov 02 01:36:49.539024 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:49.538996 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="006dbe2b-932c-4c07-9430-51b9bd3b2bb2" path="/var/lib/kubelet/pods/006dbe2b-932c-4c07-9430-51b9bd3b2bb2/volumes" Nov 02 01:36:49.539633 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:49.539605 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c3740bf-b371-41f5-9b96-0096705bf9b9" path="/var/lib/kubelet/pods/0c3740bf-b371-41f5-9b96-0096705bf9b9/volumes" Nov 02 01:36:49.540173 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:49.540154 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1699d426-179c-4719-ac8e-1d05d2e199b8" path="/var/lib/kubelet/pods/1699d426-179c-4719-ac8e-1d05d2e199b8/volumes" Nov 02 01:36:49.549328 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:49.549294 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-50537f4f-67d6cc6b87-7w7v4"] Nov 02 01:36:49.554238 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:49.554212 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-50537f4f-67d6cc6b87-7w7v4"] Nov 02 01:36:49.599318 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:49.599215 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-9a4595-5d57f7c66f-5vgsh" podStartSLOduration=5.599192471 podStartE2EDuration="5.599192471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:36:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:36:49.569781128 +0000 UTC m=+1314.190786350" watchObservedRunningTime="2026-11-02 01:36:49.599192471 +0000 UTC m=+1314.220197684" Nov 02 01:36:51.556412 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:51.556305 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e387f69-ed5c-449a-9ba5-8e03a04b235a" path="/var/lib/kubelet/pods/6e387f69-ed5c-449a-9ba5-8e03a04b235a/volumes" Nov 02 01:36:54.972957 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:54.972527 4427 patch_prober.go:29] interesting pod/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 01:36:54.972957 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:54.972630 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4" podUID="c3d179f1-c7e0-4340-a286-2efbbc44ed53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:36:55.392119 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:55.391523 4427 patch_prober.go:29] interesting pod/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 01:36:55.392119 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:55.391592 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r" podUID="c227e450-ed88-4cb4-a5fd-e88f150d14ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:36:57.236522 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:36:57.236490 4427 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@7-25610-0.service\": RecentStats: unable to find data in memory cache]" Nov 02 01:36:57.675263 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:57.675227 4427 generic.go:356] "Generic (PLEG): container finished" podID="c3d179f1-c7e0-4340-a286-2efbbc44ed53" containerID="662874da8f50b0842d9938672194dc31d4755f470d32aaa4db1518c240e7bfa5" exitCode=0 Nov 02 01:36:57.675519 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:57.675358 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4" event={"ID":"c3d179f1-c7e0-4340-a286-2efbbc44ed53","Type":"ContainerDied","Data":"662874da8f50b0842d9938672194dc31d4755f470d32aaa4db1518c240e7bfa5"} Nov 02 01:36:57.675854 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:57.675837 4427 scope.go:122] "RemoveContainer" containerID="662874da8f50b0842d9938672194dc31d4755f470d32aaa4db1518c240e7bfa5" Nov 02 01:36:57.679429 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:57.678288 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:36:57.679429 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:57.678403 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:36:57.679429 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:57.678456 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:36:57.680838 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:57.680594 4427 generic.go:356] "Generic (PLEG): container finished" podID="c227e450-ed88-4cb4-a5fd-e88f150d14ab" containerID="b572c45e63c2baf75b3cbe98f904258edf95b1dde3445abb8a01ea161ae61022" exitCode=0 Nov 02 01:36:57.680838 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:57.680697 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r" event={"ID":"c227e450-ed88-4cb4-a5fd-e88f150d14ab","Type":"ContainerDied","Data":"b572c45e63c2baf75b3cbe98f904258edf95b1dde3445abb8a01ea161ae61022"} Nov 02 01:36:57.681155 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:57.681119 4427 scope.go:122] "RemoveContainer" containerID="b572c45e63c2baf75b3cbe98f904258edf95b1dde3445abb8a01ea161ae61022" Nov 02 01:36:57.686723 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:57.686505 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:36:57.686723 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:57.686593 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:36:57.686723 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:57.686642 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:36:58.240404 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:58.235572 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl"] Nov 02 01:36:58.252637 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:58.252590 4427 cpu_manager.go:414] "containerMap: removing container" podUID="006dbe2b-932c-4c07-9430-51b9bd3b2bb2" containerName="istio-proxy" Nov 02 01:36:58.252637 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:58.252620 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="006dbe2b-932c-4c07-9430-51b9bd3b2bb2" containerName="istio-proxy" Nov 02 01:36:58.252855 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:58.252655 4427 cpu_manager.go:414] "containerMap: removing container" podUID="0c3740bf-b371-41f5-9b96-0096705bf9b9" containerName="istio-proxy" Nov 02 01:36:58.252855 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:58.252669 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c3740bf-b371-41f5-9b96-0096705bf9b9" containerName="istio-proxy" Nov 02 01:36:58.252855 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:58.252734 4427 cpu_manager.go:414] "containerMap: removing container" podUID="1699d426-179c-4719-ac8e-1d05d2e199b8" containerName="istio-proxy" Nov 02 01:36:58.252855 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:58.252742 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1699d426-179c-4719-ac8e-1d05d2e199b8" containerName="istio-proxy" Nov 02 01:36:58.252855 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:58.252809 4427 cpu_manager.go:414] "containerMap: removing container" podUID="6e387f69-ed5c-449a-9ba5-8e03a04b235a" containerName="payload-processing" Nov 02 01:36:58.252855 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:58.252825 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e387f69-ed5c-449a-9ba5-8e03a04b235a" containerName="payload-processing" Nov 02 01:36:58.253249 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:58.253222 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="006dbe2b-932c-4c07-9430-51b9bd3b2bb2" containerName="istio-proxy" Nov 02 01:36:58.253249 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:58.253245 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c3740bf-b371-41f5-9b96-0096705bf9b9" containerName="istio-proxy" Nov 02 01:36:58.256897 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:58.253280 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="1699d426-179c-4719-ac8e-1d05d2e199b8" containerName="istio-proxy" Nov 02 01:36:58.256897 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:58.253302 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e387f69-ed5c-449a-9ba5-8e03a04b235a" containerName="payload-processing" Nov 02 01:36:58.277973 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:58.277934 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl"] Nov 02 01:36:58.278861 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:58.278841 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl" Nov 02 01:36:58.289039 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:58.288983 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-00f1da-openshift-default-dockercfg-c5svk\"" Nov 02 01:36:58.435694 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:58.434600 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00825c33-7e34-4522-adf8-142b8ae3ff73-workload-socket\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl\" (UID: \"00825c33-7e34-4522-adf8-142b8ae3ff73\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl" Nov 02 01:36:58.435694 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:58.434697 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxmd4\" (UniqueName: \"kubernetes.io/projected/00825c33-7e34-4522-adf8-142b8ae3ff73-kube-api-access-hxmd4\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl\" (UID: \"00825c33-7e34-4522-adf8-142b8ae3ff73\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl" Nov 02 01:36:58.435694 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:58.434827 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00825c33-7e34-4522-adf8-142b8ae3ff73-credential-socket\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl\" (UID: \"00825c33-7e34-4522-adf8-142b8ae3ff73\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl" Nov 02 01:36:58.435694 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:58.434857 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00825c33-7e34-4522-adf8-142b8ae3ff73-istio-podinfo\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl\" (UID: \"00825c33-7e34-4522-adf8-142b8ae3ff73\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl" Nov 02 01:36:58.435694 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:58.434910 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00825c33-7e34-4522-adf8-142b8ae3ff73-istio-token\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl\" (UID: \"00825c33-7e34-4522-adf8-142b8ae3ff73\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl" Nov 02 01:36:58.435694 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:58.434988 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00825c33-7e34-4522-adf8-142b8ae3ff73-workload-certs\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl\" (UID: \"00825c33-7e34-4522-adf8-142b8ae3ff73\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl" Nov 02 01:36:58.435694 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:58.435028 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00825c33-7e34-4522-adf8-142b8ae3ff73-istio-envoy\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl\" (UID: \"00825c33-7e34-4522-adf8-142b8ae3ff73\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl" Nov 02 01:36:58.435694 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:58.435214 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00825c33-7e34-4522-adf8-142b8ae3ff73-istio-data\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl\" (UID: \"00825c33-7e34-4522-adf8-142b8ae3ff73\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl" Nov 02 01:36:58.435694 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:58.435248 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00825c33-7e34-4522-adf8-142b8ae3ff73-istiod-ca-cert\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl\" (UID: \"00825c33-7e34-4522-adf8-142b8ae3ff73\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl" Nov 02 01:36:58.537269 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:58.537189 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00825c33-7e34-4522-adf8-142b8ae3ff73-workload-socket\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl\" (UID: \"00825c33-7e34-4522-adf8-142b8ae3ff73\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl" Nov 02 01:36:58.537269 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:58.537261 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxmd4\" (UniqueName: \"kubernetes.io/projected/00825c33-7e34-4522-adf8-142b8ae3ff73-kube-api-access-hxmd4\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl\" (UID: \"00825c33-7e34-4522-adf8-142b8ae3ff73\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl" Nov 02 01:36:58.537498 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:58.537418 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00825c33-7e34-4522-adf8-142b8ae3ff73-credential-socket\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl\" (UID: \"00825c33-7e34-4522-adf8-142b8ae3ff73\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl" Nov 02 01:36:58.537498 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:58.537451 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00825c33-7e34-4522-adf8-142b8ae3ff73-istio-podinfo\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl\" (UID: \"00825c33-7e34-4522-adf8-142b8ae3ff73\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl" Nov 02 01:36:58.537576 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:58.537506 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00825c33-7e34-4522-adf8-142b8ae3ff73-istio-token\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl\" (UID: \"00825c33-7e34-4522-adf8-142b8ae3ff73\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl" Nov 02 01:36:58.537672 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:58.537649 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00825c33-7e34-4522-adf8-142b8ae3ff73-workload-socket\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl\" (UID: \"00825c33-7e34-4522-adf8-142b8ae3ff73\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl" Nov 02 01:36:58.537918 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:58.537893 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00825c33-7e34-4522-adf8-142b8ae3ff73-credential-socket\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl\" (UID: \"00825c33-7e34-4522-adf8-142b8ae3ff73\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl" Nov 02 01:36:58.538027 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:58.537981 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00825c33-7e34-4522-adf8-142b8ae3ff73-workload-certs\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl\" (UID: \"00825c33-7e34-4522-adf8-142b8ae3ff73\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl" Nov 02 01:36:58.538092 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:58.538059 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00825c33-7e34-4522-adf8-142b8ae3ff73-istio-envoy\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl\" (UID: \"00825c33-7e34-4522-adf8-142b8ae3ff73\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl" Nov 02 01:36:58.538241 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:58.538224 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00825c33-7e34-4522-adf8-142b8ae3ff73-istio-data\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl\" (UID: \"00825c33-7e34-4522-adf8-142b8ae3ff73\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl" Nov 02 01:36:58.538355 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:58.538337 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00825c33-7e34-4522-adf8-142b8ae3ff73-istiod-ca-cert\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl\" (UID: \"00825c33-7e34-4522-adf8-142b8ae3ff73\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl" Nov 02 01:36:58.538685 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:58.538255 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00825c33-7e34-4522-adf8-142b8ae3ff73-workload-certs\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl\" (UID: \"00825c33-7e34-4522-adf8-142b8ae3ff73\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl" Nov 02 01:36:58.538852 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:58.538703 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00825c33-7e34-4522-adf8-142b8ae3ff73-istio-data\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl\" (UID: \"00825c33-7e34-4522-adf8-142b8ae3ff73\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl" Nov 02 01:36:58.539557 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:58.539532 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00825c33-7e34-4522-adf8-142b8ae3ff73-istiod-ca-cert\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl\" (UID: \"00825c33-7e34-4522-adf8-142b8ae3ff73\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl" Nov 02 01:36:58.540989 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:58.540948 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00825c33-7e34-4522-adf8-142b8ae3ff73-istio-envoy\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl\" (UID: \"00825c33-7e34-4522-adf8-142b8ae3ff73\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl" Nov 02 01:36:58.541151 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:58.541126 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00825c33-7e34-4522-adf8-142b8ae3ff73-istio-podinfo\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl\" (UID: \"00825c33-7e34-4522-adf8-142b8ae3ff73\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl" Nov 02 01:36:58.552461 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:58.552416 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxmd4\" (UniqueName: \"kubernetes.io/projected/00825c33-7e34-4522-adf8-142b8ae3ff73-kube-api-access-hxmd4\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl\" (UID: \"00825c33-7e34-4522-adf8-142b8ae3ff73\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl" Nov 02 01:36:58.552677 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:58.552658 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00825c33-7e34-4522-adf8-142b8ae3ff73-istio-token\") pod \"e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl\" (UID: \"00825c33-7e34-4522-adf8-142b8ae3ff73\") " pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl" Nov 02 01:36:58.634052 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:58.634015 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl" Nov 02 01:36:58.695853 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:58.695817 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4" event={"ID":"c3d179f1-c7e0-4340-a286-2efbbc44ed53","Type":"ContainerStarted","Data":"a7ff2e236648abc871bf55a084d15da6ae70b7df221bc21f7fa2e80d0df88c64"} Nov 02 01:36:58.702441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:58.700234 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r" event={"ID":"c227e450-ed88-4cb4-a5fd-e88f150d14ab","Type":"ContainerStarted","Data":"ec2824c66fba034da4af5829a533e93ac6210f64fcc303b7204058fb3889ef4e"} Nov 02 01:36:58.707445 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:58.707132 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r" Nov 02 01:36:59.056818 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:59.056543 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl"] Nov 02 01:36:59.060423 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:59.059779 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:36:59.060423 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:59.059855 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:36:59.060423 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:59.059884 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:36:59.229973 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:59.229947 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4" Nov 02 01:36:59.230389 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:59.230352 4427 patch_prober.go:29] interesting pod/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 02 01:36:59.230491 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:59.230448 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4" podUID="c3d179f1-c7e0-4340-a286-2efbbc44ed53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 02 01:36:59.712402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:59.711545 4427 patch_prober.go:29] interesting pod/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Nov 02 01:36:59.712402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:59.711607 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r" podUID="c227e450-ed88-4cb4-a5fd-e88f150d14ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Nov 02 01:36:59.728756 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:59.728653 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl" event={"ID":"00825c33-7e34-4522-adf8-142b8ae3ff73","Type":"ContainerStarted","Data":"4500bdb77b7beb118e1cecdef9ee744975e5d25d66798934008922007f4ec916"} Nov 02 01:36:59.729512 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:59.729480 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl" event={"ID":"00825c33-7e34-4522-adf8-142b8ae3ff73","Type":"ContainerStarted","Data":"1271970a45475e3a6c770dc4b9dca769c159f2d00e9b4a4165839b11a6343945"} Nov 02 01:36:59.754805 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:36:59.754746 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl" podStartSLOduration=1.754356818 podStartE2EDuration="1.754356818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:36:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:36:59.75065977 +0000 UTC m=+1324.371665016" watchObservedRunningTime="2026-11-02 01:36:59.754356818 +0000 UTC m=+1324.375362029" Nov 02 01:37:00.246400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:00.231817 4427 patch_prober.go:29] interesting pod/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 02 01:37:00.246400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:00.231882 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4" podUID="c3d179f1-c7e0-4340-a286-2efbbc44ed53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 02 01:37:00.642450 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:00.636772 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl" Nov 02 01:37:00.642450 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:00.639561 4427 patch_prober.go:29] interesting pod/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 02 01:37:00.642450 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:00.639639 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl" podUID="00825c33-7e34-4522-adf8-142b8ae3ff73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 02 01:37:00.709838 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:00.709797 4427 patch_prober.go:29] interesting pod/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Nov 02 01:37:00.711571 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:00.711540 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r" podUID="c227e450-ed88-4cb4-a5fd-e88f150d14ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Nov 02 01:37:01.419817 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:01.409574 4427 patch_prober.go:29] interesting pod/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 02 01:37:01.419817 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:01.409635 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4" podUID="c3d179f1-c7e0-4340-a286-2efbbc44ed53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 02 01:37:01.738345 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:01.738245 4427 patch_prober.go:29] interesting pod/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 02 01:37:01.738659 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:01.738627 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl" podUID="00825c33-7e34-4522-adf8-142b8ae3ff73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 02 01:37:01.741284 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:01.741232 4427 patch_prober.go:29] interesting pod/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Nov 02 01:37:01.741532 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:01.741507 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r" podUID="c227e450-ed88-4cb4-a5fd-e88f150d14ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Nov 02 01:37:02.016471 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:37:01.931619 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:37:02.185666 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:02.162245 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9a4595-cc768d49f-fb6l8" Nov 02 01:37:02.200400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:02.162367 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-9a4595-5d57f7c66f-5vgsh" Nov 02 01:37:02.325463 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:02.268978 4427 patch_prober.go:29] interesting pod/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 02 01:37:02.325463 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:02.269062 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4" podUID="c3d179f1-c7e0-4340-a286-2efbbc44ed53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 02 01:37:02.645930 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:02.645896 4427 patch_prober.go:29] interesting pod/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 02 01:37:02.646567 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:02.646521 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl" podUID="00825c33-7e34-4522-adf8-142b8ae3ff73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 02 01:37:02.746024 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:02.745964 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" podUID="d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:37:02.746954 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:02.746900 4427 patch_prober.go:29] interesting pod/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Nov 02 01:37:02.747129 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:02.746967 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r" podUID="c227e450-ed88-4cb4-a5fd-e88f150d14ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Nov 02 01:37:03.243399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:03.234188 4427 patch_prober.go:29] interesting pod/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 02 01:37:03.243399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:03.234253 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4" podUID="c3d179f1-c7e0-4340-a286-2efbbc44ed53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 02 01:37:03.652272 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:03.652231 4427 patch_prober.go:29] interesting pod/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 02 01:37:03.678518 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:03.678464 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl" podUID="00825c33-7e34-4522-adf8-142b8ae3ff73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 02 01:37:03.712422 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:03.712110 4427 patch_prober.go:29] interesting pod/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Nov 02 01:37:03.712422 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:03.712175 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r" podUID="c227e450-ed88-4cb4-a5fd-e88f150d14ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Nov 02 01:37:04.243402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:04.240152 4427 patch_prober.go:29] interesting pod/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 02 01:37:04.243402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:04.240216 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4" podUID="c3d179f1-c7e0-4340-a286-2efbbc44ed53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 02 01:37:04.671451 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:04.651526 4427 patch_prober.go:29] interesting pod/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 02 01:37:04.671451 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:04.651589 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl" podUID="00825c33-7e34-4522-adf8-142b8ae3ff73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 02 01:37:04.833903 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:04.748826 4427 patch_prober.go:29] interesting pod/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Nov 02 01:37:04.833903 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:04.748890 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r" podUID="c227e450-ed88-4cb4-a5fd-e88f150d14ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Nov 02 01:37:05.343273 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:05.343229 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4" Nov 02 01:37:05.344722 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:05.344569 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4" Nov 02 01:37:05.347142 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:05.347052 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4" Nov 02 01:37:05.635428 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:05.635328 4427 patch_prober.go:29] interesting pod/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 02 01:37:05.635816 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:05.635703 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl" podUID="00825c33-7e34-4522-adf8-142b8ae3ff73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 02 01:37:05.713128 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:05.713092 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r" Nov 02 01:37:05.713647 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:05.713495 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r" Nov 02 01:37:05.714828 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:05.714807 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r" Nov 02 01:37:06.219781 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:06.219736 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9a4595-554977c4cb-vhq5g"] Nov 02 01:37:06.226315 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:06.226286 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a4595-554977c4cb-vhq5g" Nov 02 01:37:06.235626 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:06.235587 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-9a4595-serving-cert\"" Nov 02 01:37:06.243363 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:06.242553 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9a4595-554977c4cb-vhq5g"] Nov 02 01:37:06.360708 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:06.360674 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wzpw\" (UniqueName: \"kubernetes.io/projected/57445e45-2af1-475b-a32b-e8d88a5cdc54-kube-api-access-5wzpw\") pod \"maas-api-e2e-inf-b-9a4595-554977c4cb-vhq5g\" (UID: \"57445e45-2af1-475b-a32b-e8d88a5cdc54\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a4595-554977c4cb-vhq5g" Nov 02 01:37:06.360975 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:06.360954 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57445e45-2af1-475b-a32b-e8d88a5cdc54-maas-api-tls\") pod \"maas-api-e2e-inf-b-9a4595-554977c4cb-vhq5g\" (UID: \"57445e45-2af1-475b-a32b-e8d88a5cdc54\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a4595-554977c4cb-vhq5g" Nov 02 01:37:06.361149 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:06.361115 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57445e45-2af1-475b-a32b-e8d88a5cdc54-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9a4595-554977c4cb-vhq5g\" (UID: \"57445e45-2af1-475b-a32b-e8d88a5cdc54\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a4595-554977c4cb-vhq5g" Nov 02 01:37:06.361216 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:06.361201 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57445e45-2af1-475b-a32b-e8d88a5cdc54-metrics-certs\") pod \"maas-api-e2e-inf-b-9a4595-554977c4cb-vhq5g\" (UID: \"57445e45-2af1-475b-a32b-e8d88a5cdc54\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a4595-554977c4cb-vhq5g" Nov 02 01:37:06.467160 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:06.467118 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57445e45-2af1-475b-a32b-e8d88a5cdc54-maas-api-tls\") pod \"maas-api-e2e-inf-b-9a4595-554977c4cb-vhq5g\" (UID: \"57445e45-2af1-475b-a32b-e8d88a5cdc54\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a4595-554977c4cb-vhq5g" Nov 02 01:37:06.467368 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:06.467281 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57445e45-2af1-475b-a32b-e8d88a5cdc54-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9a4595-554977c4cb-vhq5g\" (UID: \"57445e45-2af1-475b-a32b-e8d88a5cdc54\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a4595-554977c4cb-vhq5g" Nov 02 01:37:06.467368 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:06.467367 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57445e45-2af1-475b-a32b-e8d88a5cdc54-metrics-certs\") pod \"maas-api-e2e-inf-b-9a4595-554977c4cb-vhq5g\" (UID: \"57445e45-2af1-475b-a32b-e8d88a5cdc54\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a4595-554977c4cb-vhq5g" Nov 02 01:37:06.467564 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:06.467532 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wzpw\" (UniqueName: \"kubernetes.io/projected/57445e45-2af1-475b-a32b-e8d88a5cdc54-kube-api-access-5wzpw\") pod \"maas-api-e2e-inf-b-9a4595-554977c4cb-vhq5g\" (UID: \"57445e45-2af1-475b-a32b-e8d88a5cdc54\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a4595-554977c4cb-vhq5g" Nov 02 01:37:06.470354 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:06.470273 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57445e45-2af1-475b-a32b-e8d88a5cdc54-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9a4595-554977c4cb-vhq5g\" (UID: \"57445e45-2af1-475b-a32b-e8d88a5cdc54\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a4595-554977c4cb-vhq5g" Nov 02 01:37:06.481065 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:06.481024 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57445e45-2af1-475b-a32b-e8d88a5cdc54-maas-api-tls\") pod \"maas-api-e2e-inf-b-9a4595-554977c4cb-vhq5g\" (UID: \"57445e45-2af1-475b-a32b-e8d88a5cdc54\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a4595-554977c4cb-vhq5g" Nov 02 01:37:06.481907 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:06.481872 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57445e45-2af1-475b-a32b-e8d88a5cdc54-metrics-certs\") pod \"maas-api-e2e-inf-b-9a4595-554977c4cb-vhq5g\" (UID: \"57445e45-2af1-475b-a32b-e8d88a5cdc54\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a4595-554977c4cb-vhq5g" Nov 02 01:37:06.484008 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:06.483988 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wzpw\" (UniqueName: \"kubernetes.io/projected/57445e45-2af1-475b-a32b-e8d88a5cdc54-kube-api-access-5wzpw\") pod \"maas-api-e2e-inf-b-9a4595-554977c4cb-vhq5g\" (UID: \"57445e45-2af1-475b-a32b-e8d88a5cdc54\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a4595-554977c4cb-vhq5g" Nov 02 01:37:06.561109 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:06.561074 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a4595-554977c4cb-vhq5g" Nov 02 01:37:06.643466 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:06.643090 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl" Nov 02 01:37:06.647443 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:06.645131 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl" Nov 02 01:37:06.647443 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:06.646614 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl" Nov 02 01:37:06.885276 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:06.885225 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9a4595-554977c4cb-vhq5g"] Nov 02 01:37:07.178156 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:07.176428 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a4595-554977c4cb-vhq5g" event={"ID":"57445e45-2af1-475b-a32b-e8d88a5cdc54","Type":"ContainerStarted","Data":"79190fe346b45022db532e3e229fa88cae4290a82762cae4be18470db486c2b9"} Nov 02 01:37:08.189288 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:08.189245 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a4595-554977c4cb-vhq5g" event={"ID":"57445e45-2af1-475b-a32b-e8d88a5cdc54","Type":"ContainerStarted","Data":"d7affb1576cabd70041a3aae992bad6e04a89aa969e6749d094677fa63ac66d2"} Nov 02 01:37:08.189901 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:08.189873 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a4595-554977c4cb-vhq5g" Nov 02 01:37:08.216399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:08.213077 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a4595-554977c4cb-vhq5g" podStartSLOduration=1.634358979 podStartE2EDuration="2.213054921s" totalImagesPullingTime="578.695942ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:37:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:37:08.210126203 +0000 UTC m=+1332.831131420" watchObservedRunningTime="2026-11-02 01:37:08.213054921 +0000 UTC m=+1332.834060135" Nov 02 01:37:14.203230 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:14.203199 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a4595-554977c4cb-vhq5g" Nov 02 01:37:31.627194 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:37:31.627152 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:37:41.190939 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:41.188444 4427 patch_prober.go:29] interesting pod/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": read tcp 10.133.0.2:45294->10.133.0.119:15021: read: connection reset by peer" start-of-body= Nov 02 01:37:41.190939 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:41.188522 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7" podUID="273e0a10-0ba6-41bb-b8db-1ff672905957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": read tcp 10.133.0.2:45294->10.133.0.119:15021: read: connection reset by peer" Nov 02 01:37:41.830501 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:41.830469 4427 generic.go:356] "Generic (PLEG): container finished" podID="273e0a10-0ba6-41bb-b8db-1ff672905957" containerID="31481e93dfdddb0e8abe4b5f38d64987489513b46bdc3259dcacd8fa44b152ba" exitCode=0 Nov 02 01:37:41.830675 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:41.830525 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7" event={"ID":"273e0a10-0ba6-41bb-b8db-1ff672905957","Type":"ContainerDied","Data":"31481e93dfdddb0e8abe4b5f38d64987489513b46bdc3259dcacd8fa44b152ba"} Nov 02 01:37:41.831025 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:41.831011 4427 scope.go:122] "RemoveContainer" containerID="31481e93dfdddb0e8abe4b5f38d64987489513b46bdc3259dcacd8fa44b152ba" Nov 02 01:37:41.833134 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:41.833103 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:37:41.833237 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:41.833181 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:37:41.833237 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:41.833216 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:37:42.736038 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:42.736005 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n"] Nov 02 01:37:42.741207 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:42.741182 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n" Nov 02 01:37:42.752580 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:42.752184 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-12e7cb-openshift-default-dockercfg-vkhn5\"" Nov 02 01:37:42.767595 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:42.767440 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n"] Nov 02 01:37:42.812803 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:42.812768 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/331a730d-c524-4f68-9360-25e29df1ffb4-istio-envoy\") pod \"e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n\" (UID: \"331a730d-c524-4f68-9360-25e29df1ffb4\") " pod="openshift-ingress/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n" Nov 02 01:37:42.813000 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:42.812821 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/331a730d-c524-4f68-9360-25e29df1ffb4-istio-token\") pod \"e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n\" (UID: \"331a730d-c524-4f68-9360-25e29df1ffb4\") " pod="openshift-ingress/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n" Nov 02 01:37:42.813071 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:42.812987 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl9c4\" (UniqueName: \"kubernetes.io/projected/331a730d-c524-4f68-9360-25e29df1ffb4-kube-api-access-fl9c4\") pod \"e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n\" (UID: \"331a730d-c524-4f68-9360-25e29df1ffb4\") " pod="openshift-ingress/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n" Nov 02 01:37:42.813071 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:42.813051 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/331a730d-c524-4f68-9360-25e29df1ffb4-workload-socket\") pod \"e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n\" (UID: \"331a730d-c524-4f68-9360-25e29df1ffb4\") " pod="openshift-ingress/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n" Nov 02 01:37:42.813167 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:42.813119 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/331a730d-c524-4f68-9360-25e29df1ffb4-istiod-ca-cert\") pod \"e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n\" (UID: \"331a730d-c524-4f68-9360-25e29df1ffb4\") " pod="openshift-ingress/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n" Nov 02 01:37:42.813233 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:42.813216 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/331a730d-c524-4f68-9360-25e29df1ffb4-istio-podinfo\") pod \"e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n\" (UID: \"331a730d-c524-4f68-9360-25e29df1ffb4\") " pod="openshift-ingress/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n" Nov 02 01:37:42.813311 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:42.813293 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/331a730d-c524-4f68-9360-25e29df1ffb4-credential-socket\") pod \"e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n\" (UID: \"331a730d-c524-4f68-9360-25e29df1ffb4\") " pod="openshift-ingress/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n" Nov 02 01:37:42.813446 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:42.813429 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/331a730d-c524-4f68-9360-25e29df1ffb4-istio-data\") pod \"e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n\" (UID: \"331a730d-c524-4f68-9360-25e29df1ffb4\") " pod="openshift-ingress/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n" Nov 02 01:37:42.813519 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:42.813492 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/331a730d-c524-4f68-9360-25e29df1ffb4-workload-certs\") pod \"e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n\" (UID: \"331a730d-c524-4f68-9360-25e29df1ffb4\") " pod="openshift-ingress/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n" Nov 02 01:37:42.935398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:42.931540 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/331a730d-c524-4f68-9360-25e29df1ffb4-istio-podinfo\") pod \"e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n\" (UID: \"331a730d-c524-4f68-9360-25e29df1ffb4\") " pod="openshift-ingress/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n" Nov 02 01:37:42.935398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:42.931626 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/331a730d-c524-4f68-9360-25e29df1ffb4-credential-socket\") pod \"e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n\" (UID: \"331a730d-c524-4f68-9360-25e29df1ffb4\") " pod="openshift-ingress/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n" Nov 02 01:37:42.935398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:42.931724 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/331a730d-c524-4f68-9360-25e29df1ffb4-istio-data\") pod \"e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n\" (UID: \"331a730d-c524-4f68-9360-25e29df1ffb4\") " pod="openshift-ingress/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n" Nov 02 01:37:42.935398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:42.931825 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/331a730d-c524-4f68-9360-25e29df1ffb4-workload-certs\") pod \"e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n\" (UID: \"331a730d-c524-4f68-9360-25e29df1ffb4\") " pod="openshift-ingress/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n" Nov 02 01:37:42.935398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:42.932011 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/331a730d-c524-4f68-9360-25e29df1ffb4-istio-envoy\") pod \"e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n\" (UID: \"331a730d-c524-4f68-9360-25e29df1ffb4\") " pod="openshift-ingress/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n" Nov 02 01:37:42.935398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:42.932056 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/331a730d-c524-4f68-9360-25e29df1ffb4-istio-token\") pod \"e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n\" (UID: \"331a730d-c524-4f68-9360-25e29df1ffb4\") " pod="openshift-ingress/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n" Nov 02 01:37:42.935398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:42.932155 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl9c4\" (UniqueName: \"kubernetes.io/projected/331a730d-c524-4f68-9360-25e29df1ffb4-kube-api-access-fl9c4\") pod \"e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n\" (UID: \"331a730d-c524-4f68-9360-25e29df1ffb4\") " pod="openshift-ingress/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n" Nov 02 01:37:42.935398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:42.932208 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/331a730d-c524-4f68-9360-25e29df1ffb4-workload-socket\") pod \"e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n\" (UID: \"331a730d-c524-4f68-9360-25e29df1ffb4\") " pod="openshift-ingress/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n" Nov 02 01:37:42.935398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:42.932232 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/331a730d-c524-4f68-9360-25e29df1ffb4-istiod-ca-cert\") pod \"e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n\" (UID: \"331a730d-c524-4f68-9360-25e29df1ffb4\") " pod="openshift-ingress/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n" Nov 02 01:37:42.953396 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:42.945004 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/331a730d-c524-4f68-9360-25e29df1ffb4-workload-certs\") pod \"e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n\" (UID: \"331a730d-c524-4f68-9360-25e29df1ffb4\") " pod="openshift-ingress/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n" Nov 02 01:37:42.953396 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:42.946668 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/331a730d-c524-4f68-9360-25e29df1ffb4-credential-socket\") pod \"e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n\" (UID: \"331a730d-c524-4f68-9360-25e29df1ffb4\") " pod="openshift-ingress/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n" Nov 02 01:37:42.953396 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:42.946689 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7" event={"ID":"273e0a10-0ba6-41bb-b8db-1ff672905957","Type":"ContainerStarted","Data":"77854ebe0744657b26b6612e13f07e1bb749c776817ca86199013bbf26894742"} Nov 02 01:37:42.953396 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:42.946953 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/331a730d-c524-4f68-9360-25e29df1ffb4-istio-data\") pod \"e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n\" (UID: \"331a730d-c524-4f68-9360-25e29df1ffb4\") " pod="openshift-ingress/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n" Nov 02 01:37:42.953396 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:42.948189 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/331a730d-c524-4f68-9360-25e29df1ffb4-workload-socket\") pod \"e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n\" (UID: \"331a730d-c524-4f68-9360-25e29df1ffb4\") " pod="openshift-ingress/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n" Nov 02 01:37:42.953817 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:42.953544 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/331a730d-c524-4f68-9360-25e29df1ffb4-istiod-ca-cert\") pod \"e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n\" (UID: \"331a730d-c524-4f68-9360-25e29df1ffb4\") " pod="openshift-ingress/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n" Nov 02 01:37:43.012454 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:43.012266 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/331a730d-c524-4f68-9360-25e29df1ffb4-istio-podinfo\") pod \"e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n\" (UID: \"331a730d-c524-4f68-9360-25e29df1ffb4\") " pod="openshift-ingress/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n" Nov 02 01:37:43.027495 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:43.023606 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl9c4\" (UniqueName: \"kubernetes.io/projected/331a730d-c524-4f68-9360-25e29df1ffb4-kube-api-access-fl9c4\") pod \"e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n\" (UID: \"331a730d-c524-4f68-9360-25e29df1ffb4\") " pod="openshift-ingress/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n" Nov 02 01:37:43.027495 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:43.025623 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/331a730d-c524-4f68-9360-25e29df1ffb4-istio-envoy\") pod \"e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n\" (UID: \"331a730d-c524-4f68-9360-25e29df1ffb4\") " pod="openshift-ingress/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n" Nov 02 01:37:43.076399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:43.069920 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/331a730d-c524-4f68-9360-25e29df1ffb4-istio-token\") pod \"e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n\" (UID: \"331a730d-c524-4f68-9360-25e29df1ffb4\") " pod="openshift-ingress/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n" Nov 02 01:37:43.094404 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:43.083677 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n" Nov 02 01:37:43.675162 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:43.675130 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7" Nov 02 01:37:43.694593 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:43.694031 4427 patch_prober.go:29] interesting pod/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Nov 02 01:37:43.694593 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:43.694096 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7" podUID="273e0a10-0ba6-41bb-b8db-1ff672905957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Nov 02 01:37:43.746362 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:43.746255 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n"] Nov 02 01:37:43.782655 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:43.765967 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:37:43.782655 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:43.766060 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:37:43.782655 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:43.766105 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:37:44.005619 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:44.003641 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n" event={"ID":"331a730d-c524-4f68-9360-25e29df1ffb4","Type":"ContainerStarted","Data":"57486b9e30ae367c16b1c917c510c2aad33b96025e35d27ba63d680d0d2e82ed"} Nov 02 01:37:44.005619 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:44.003751 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n" event={"ID":"331a730d-c524-4f68-9360-25e29df1ffb4","Type":"ContainerStarted","Data":"d84bd8ef7dc9f9d075dc58d9d1be3e2cff2fd7966cf092e0cda9c59fab6dc7cf"} Nov 02 01:37:44.035981 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:44.035921 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n" podStartSLOduration=2.035903503 podStartE2EDuration="2.035903503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:37:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:37:44.031065601 +0000 UTC m=+1368.652070812" watchObservedRunningTime="2026-11-02 01:37:44.035903503 +0000 UTC m=+1368.656908715" Nov 02 01:37:44.093413 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:44.091218 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n" Nov 02 01:37:44.098528 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:44.095214 4427 patch_prober.go:29] interesting pod/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 02 01:37:44.098528 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:44.095269 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n" podUID="331a730d-c524-4f68-9360-25e29df1ffb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 02 01:37:44.671402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:44.667957 4427 patch_prober.go:29] interesting pod/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Nov 02 01:37:44.671402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:44.668154 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7" podUID="273e0a10-0ba6-41bb-b8db-1ff672905957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Nov 02 01:37:45.095450 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:45.095326 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n" Nov 02 01:37:45.660736 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:45.660699 4427 patch_prober.go:29] interesting pod/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Nov 02 01:37:45.661052 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:45.661022 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7" podUID="273e0a10-0ba6-41bb-b8db-1ff672905957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Nov 02 01:37:46.021539 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:46.021414 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n" Nov 02 01:37:46.022628 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:46.022604 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n" Nov 02 01:37:46.682396 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:46.682329 4427 patch_prober.go:29] interesting pod/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Nov 02 01:37:46.682976 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:46.682407 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7" podUID="273e0a10-0ba6-41bb-b8db-1ff672905957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Nov 02 01:37:47.667398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:47.666733 4427 patch_prober.go:29] interesting pod/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Nov 02 01:37:47.667398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:47.666798 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7" podUID="273e0a10-0ba6-41bb-b8db-1ff672905957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Nov 02 01:37:48.684903 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:48.684861 4427 patch_prober.go:29] interesting pod/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Nov 02 01:37:48.685445 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:48.684947 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7" podUID="273e0a10-0ba6-41bb-b8db-1ff672905957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Nov 02 01:37:49.687112 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:49.685059 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7" Nov 02 01:37:49.687112 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:49.686957 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7" Nov 02 01:37:49.690041 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:49.690018 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7" Nov 02 01:37:51.606090 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:51.606043 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp"] Nov 02 01:37:51.611494 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:51.611461 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp" Nov 02 01:37:51.621532 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:51.621496 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp"] Nov 02 01:37:51.684821 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:51.684791 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1371b4cc-965f-4e8c-a482-6f2e4c76282e-istio-podinfo\") pod \"e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp\" (UID: \"1371b4cc-965f-4e8c-a482-6f2e4c76282e\") " pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp" Nov 02 01:37:51.684995 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:51.684843 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1371b4cc-965f-4e8c-a482-6f2e4c76282e-istio-envoy\") pod \"e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp\" (UID: \"1371b4cc-965f-4e8c-a482-6f2e4c76282e\") " pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp" Nov 02 01:37:51.685040 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:51.685004 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1371b4cc-965f-4e8c-a482-6f2e4c76282e-credential-socket\") pod \"e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp\" (UID: \"1371b4cc-965f-4e8c-a482-6f2e4c76282e\") " pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp" Nov 02 01:37:51.685077 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:51.685051 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1371b4cc-965f-4e8c-a482-6f2e4c76282e-workload-certs\") pod \"e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp\" (UID: \"1371b4cc-965f-4e8c-a482-6f2e4c76282e\") " pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp" Nov 02 01:37:51.685138 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:51.685126 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1371b4cc-965f-4e8c-a482-6f2e4c76282e-workload-socket\") pod \"e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp\" (UID: \"1371b4cc-965f-4e8c-a482-6f2e4c76282e\") " pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp" Nov 02 01:37:51.685181 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:51.685171 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1371b4cc-965f-4e8c-a482-6f2e4c76282e-istiod-ca-cert\") pod \"e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp\" (UID: \"1371b4cc-965f-4e8c-a482-6f2e4c76282e\") " pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp" Nov 02 01:37:51.685309 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:51.685293 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bntfl\" (UniqueName: \"kubernetes.io/projected/1371b4cc-965f-4e8c-a482-6f2e4c76282e-kube-api-access-bntfl\") pod \"e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp\" (UID: \"1371b4cc-965f-4e8c-a482-6f2e4c76282e\") " pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp" Nov 02 01:37:51.685512 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:51.685496 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1371b4cc-965f-4e8c-a482-6f2e4c76282e-istio-data\") pod \"e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp\" (UID: \"1371b4cc-965f-4e8c-a482-6f2e4c76282e\") " pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp" Nov 02 01:37:51.685551 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:51.685526 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1371b4cc-965f-4e8c-a482-6f2e4c76282e-istio-token\") pod \"e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp\" (UID: \"1371b4cc-965f-4e8c-a482-6f2e4c76282e\") " pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp" Nov 02 01:37:51.695497 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:51.695465 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-addad9-openshift-default-dockercfg-km8ng\"" Nov 02 01:37:51.789405 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:51.788500 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1371b4cc-965f-4e8c-a482-6f2e4c76282e-credential-socket\") pod \"e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp\" (UID: \"1371b4cc-965f-4e8c-a482-6f2e4c76282e\") " pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp" Nov 02 01:37:51.789405 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:51.788611 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1371b4cc-965f-4e8c-a482-6f2e4c76282e-workload-certs\") pod \"e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp\" (UID: \"1371b4cc-965f-4e8c-a482-6f2e4c76282e\") " pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp" Nov 02 01:37:51.789405 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:51.788682 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1371b4cc-965f-4e8c-a482-6f2e4c76282e-workload-socket\") pod \"e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp\" (UID: \"1371b4cc-965f-4e8c-a482-6f2e4c76282e\") " pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp" Nov 02 01:37:51.789405 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:51.788741 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1371b4cc-965f-4e8c-a482-6f2e4c76282e-istiod-ca-cert\") pod \"e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp\" (UID: \"1371b4cc-965f-4e8c-a482-6f2e4c76282e\") " pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp" Nov 02 01:37:51.789405 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:51.788844 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bntfl\" (UniqueName: \"kubernetes.io/projected/1371b4cc-965f-4e8c-a482-6f2e4c76282e-kube-api-access-bntfl\") pod \"e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp\" (UID: \"1371b4cc-965f-4e8c-a482-6f2e4c76282e\") " pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp" Nov 02 01:37:51.789405 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:51.789017 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1371b4cc-965f-4e8c-a482-6f2e4c76282e-istio-data\") pod \"e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp\" (UID: \"1371b4cc-965f-4e8c-a482-6f2e4c76282e\") " pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp" Nov 02 01:37:51.789405 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:51.789062 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1371b4cc-965f-4e8c-a482-6f2e4c76282e-istio-token\") pod \"e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp\" (UID: \"1371b4cc-965f-4e8c-a482-6f2e4c76282e\") " pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp" Nov 02 01:37:51.789405 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:51.789116 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1371b4cc-965f-4e8c-a482-6f2e4c76282e-istio-podinfo\") pod \"e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp\" (UID: \"1371b4cc-965f-4e8c-a482-6f2e4c76282e\") " pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp" Nov 02 01:37:51.789405 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:51.789171 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1371b4cc-965f-4e8c-a482-6f2e4c76282e-istio-envoy\") pod \"e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp\" (UID: \"1371b4cc-965f-4e8c-a482-6f2e4c76282e\") " pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp" Nov 02 01:37:51.793412 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:51.790184 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1371b4cc-965f-4e8c-a482-6f2e4c76282e-workload-certs\") pod \"e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp\" (UID: \"1371b4cc-965f-4e8c-a482-6f2e4c76282e\") " pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp" Nov 02 01:37:51.793412 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:51.790945 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1371b4cc-965f-4e8c-a482-6f2e4c76282e-credential-socket\") pod \"e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp\" (UID: \"1371b4cc-965f-4e8c-a482-6f2e4c76282e\") " pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp" Nov 02 01:37:51.793412 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:51.791417 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1371b4cc-965f-4e8c-a482-6f2e4c76282e-istiod-ca-cert\") pod \"e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp\" (UID: \"1371b4cc-965f-4e8c-a482-6f2e4c76282e\") " pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp" Nov 02 01:37:51.793412 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:51.791604 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1371b4cc-965f-4e8c-a482-6f2e4c76282e-istio-data\") pod \"e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp\" (UID: \"1371b4cc-965f-4e8c-a482-6f2e4c76282e\") " pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp" Nov 02 01:37:51.793412 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:51.791552 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1371b4cc-965f-4e8c-a482-6f2e4c76282e-workload-socket\") pod \"e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp\" (UID: \"1371b4cc-965f-4e8c-a482-6f2e4c76282e\") " pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp" Nov 02 01:37:51.806081 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:51.806052 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1371b4cc-965f-4e8c-a482-6f2e4c76282e-istio-envoy\") pod \"e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp\" (UID: \"1371b4cc-965f-4e8c-a482-6f2e4c76282e\") " pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp" Nov 02 01:37:51.807685 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:51.806520 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1371b4cc-965f-4e8c-a482-6f2e4c76282e-istio-podinfo\") pod \"e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp\" (UID: \"1371b4cc-965f-4e8c-a482-6f2e4c76282e\") " pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp" Nov 02 01:37:51.810099 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:51.810050 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1371b4cc-965f-4e8c-a482-6f2e4c76282e-istio-token\") pod \"e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp\" (UID: \"1371b4cc-965f-4e8c-a482-6f2e4c76282e\") " pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp" Nov 02 01:37:51.810917 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:51.810888 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bntfl\" (UniqueName: \"kubernetes.io/projected/1371b4cc-965f-4e8c-a482-6f2e4c76282e-kube-api-access-bntfl\") pod \"e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp\" (UID: \"1371b4cc-965f-4e8c-a482-6f2e4c76282e\") " pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp" Nov 02 01:37:51.938957 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:51.938919 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp" Nov 02 01:37:52.397187 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:52.388985 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp"] Nov 02 01:37:52.465390 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:52.465310 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:37:52.465755 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:52.465726 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:37:52.465995 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:52.465973 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:37:53.142520 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:53.142472 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp" event={"ID":"1371b4cc-965f-4e8c-a482-6f2e4c76282e","Type":"ContainerStarted","Data":"5faafcb73087f7b6d8a1811f8e752f1a90e2325962d4ed4041af215da5984334"} Nov 02 01:37:53.143180 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:53.142537 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp" event={"ID":"1371b4cc-965f-4e8c-a482-6f2e4c76282e","Type":"ContainerStarted","Data":"dbca344df7ba3a449d9679a4ab32dc7b5602b20e3031bf585baff6d602a9358e"} Nov 02 01:37:53.173554 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:53.171099 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp" podStartSLOduration=2.171077969 podStartE2EDuration="2.171077969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:37:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:37:53.16648835 +0000 UTC m=+1377.787493575" watchObservedRunningTime="2026-11-02 01:37:53.171077969 +0000 UTC m=+1377.792083183" Nov 02 01:37:53.939456 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:53.939421 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp" Nov 02 01:37:53.941553 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:53.941523 4427 patch_prober.go:29] interesting pod/e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 02 01:37:53.941694 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:53.941580 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp" podUID="1371b4cc-965f-4e8c-a482-6f2e4c76282e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 02 01:37:54.939515 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:54.939481 4427 patch_prober.go:29] interesting pod/e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 02 01:37:54.940027 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:54.939539 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp" podUID="1371b4cc-965f-4e8c-a482-6f2e4c76282e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 02 01:37:55.946069 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:55.942967 4427 patch_prober.go:29] interesting pod/e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 02 01:37:55.952576 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:55.952526 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp" podUID="1371b4cc-965f-4e8c-a482-6f2e4c76282e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 02 01:37:56.939881 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:56.939845 4427 patch_prober.go:29] interesting pod/e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 02 01:37:56.940064 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:56.939896 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp" podUID="1371b4cc-965f-4e8c-a482-6f2e4c76282e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 02 01:37:57.940022 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:57.939986 4427 patch_prober.go:29] interesting pod/e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 02 01:37:57.940470 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:57.940039 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp" podUID="1371b4cc-965f-4e8c-a482-6f2e4c76282e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 02 01:37:58.939781 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:58.939745 4427 patch_prober.go:29] interesting pod/e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 02 01:37:58.939989 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:58.939803 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp" podUID="1371b4cc-965f-4e8c-a482-6f2e4c76282e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 02 01:37:59.937318 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:59.937281 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4"] Nov 02 01:37:59.938185 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:59.938154 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4" podUID="f8eafff6-528c-4e61-adff-8aed03dea982" containerName="istio-proxy" containerID="cri-o://a4acdb033cd6d67a4f1297a7d368d8477e0472d284b6ecc3984a7adf9d07717d" gracePeriod=30 Nov 02 01:37:59.946743 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:59.946719 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp" Nov 02 01:37:59.947121 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:59.947035 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp" Nov 02 01:37:59.964385 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:59.964336 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp" Nov 02 01:37:59.986771 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:59.986718 4427 patch_prober.go:29] interesting pod/e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 01:37:59.987039 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:37:59.986792 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4" podUID="f8eafff6-528c-4e61-adff-8aed03dea982" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 01:38:01.587345 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:38:01.587310 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:38:05.270061 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:05.270034 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4" Nov 02 01:38:05.336033 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:05.336002 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8eafff6-528c-4e61-adff-8aed03dea982-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8eafff6-528c-4e61-adff-8aed03dea982-istio-data\") pod \"f8eafff6-528c-4e61-adff-8aed03dea982\" (UID: \"f8eafff6-528c-4e61-adff-8aed03dea982\") " Nov 02 01:38:05.336218 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:05.336046 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8eafff6-528c-4e61-adff-8aed03dea982-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8eafff6-528c-4e61-adff-8aed03dea982-workload-socket\") pod \"f8eafff6-528c-4e61-adff-8aed03dea982\" (UID: \"f8eafff6-528c-4e61-adff-8aed03dea982\") " Nov 02 01:38:05.336218 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:05.336111 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8eafff6-528c-4e61-adff-8aed03dea982-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8eafff6-528c-4e61-adff-8aed03dea982-credential-socket\") pod \"f8eafff6-528c-4e61-adff-8aed03dea982\" (UID: \"f8eafff6-528c-4e61-adff-8aed03dea982\") " Nov 02 01:38:05.336218 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:05.336157 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8eafff6-528c-4e61-adff-8aed03dea982-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8eafff6-528c-4e61-adff-8aed03dea982-istio-envoy\") pod \"f8eafff6-528c-4e61-adff-8aed03dea982\" (UID: \"f8eafff6-528c-4e61-adff-8aed03dea982\") " Nov 02 01:38:05.336218 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:05.336186 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8eafff6-528c-4e61-adff-8aed03dea982-istio-token\" (UniqueName: \"kubernetes.io/projected/f8eafff6-528c-4e61-adff-8aed03dea982-istio-token\") pod \"f8eafff6-528c-4e61-adff-8aed03dea982\" (UID: \"f8eafff6-528c-4e61-adff-8aed03dea982\") " Nov 02 01:38:05.336478 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:05.336235 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8eafff6-528c-4e61-adff-8aed03dea982-kube-api-access-m86xd\" (UniqueName: \"kubernetes.io/projected/f8eafff6-528c-4e61-adff-8aed03dea982-kube-api-access-m86xd\") pod \"f8eafff6-528c-4e61-adff-8aed03dea982\" (UID: \"f8eafff6-528c-4e61-adff-8aed03dea982\") " Nov 02 01:38:05.336478 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:05.336313 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8eafff6-528c-4e61-adff-8aed03dea982-istio-data" pod "f8eafff6-528c-4e61-adff-8aed03dea982" (UID: "f8eafff6-528c-4e61-adff-8aed03dea982"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:38:05.336478 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:05.336385 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8eafff6-528c-4e61-adff-8aed03dea982-workload-socket" pod "f8eafff6-528c-4e61-adff-8aed03dea982" (UID: "f8eafff6-528c-4e61-adff-8aed03dea982"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:38:05.336478 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:05.336462 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8eafff6-528c-4e61-adff-8aed03dea982-credential-socket" pod "f8eafff6-528c-4e61-adff-8aed03dea982" (UID: "f8eafff6-528c-4e61-adff-8aed03dea982"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:38:05.336696 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:05.336678 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8eafff6-528c-4e61-adff-8aed03dea982-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8eafff6-528c-4e61-adff-8aed03dea982-istiod-ca-cert\") pod \"f8eafff6-528c-4e61-adff-8aed03dea982\" (UID: \"f8eafff6-528c-4e61-adff-8aed03dea982\") " Nov 02 01:38:05.336832 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:05.336807 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8eafff6-528c-4e61-adff-8aed03dea982-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8eafff6-528c-4e61-adff-8aed03dea982-workload-certs\") pod \"f8eafff6-528c-4e61-adff-8aed03dea982\" (UID: \"f8eafff6-528c-4e61-adff-8aed03dea982\") " Nov 02 01:38:05.337020 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:05.336845 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f8eafff6-528c-4e61-adff-8aed03dea982-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8eafff6-528c-4e61-adff-8aed03dea982-istio-podinfo\") pod \"f8eafff6-528c-4e61-adff-8aed03dea982\" (UID: \"f8eafff6-528c-4e61-adff-8aed03dea982\") " Nov 02 01:38:05.337261 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:05.337238 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8eafff6-528c-4e61-adff-8aed03dea982-workload-certs" pod "f8eafff6-528c-4e61-adff-8aed03dea982" (UID: "f8eafff6-528c-4e61-adff-8aed03dea982"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:38:05.337329 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:05.337282 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8eafff6-528c-4e61-adff-8aed03dea982-istiod-ca-cert" pod "f8eafff6-528c-4e61-adff-8aed03dea982" (UID: "f8eafff6-528c-4e61-adff-8aed03dea982"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:38:05.338075 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:05.338051 4427 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8eafff6-528c-4e61-adff-8aed03dea982-istiod-ca-cert\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:38:05.338181 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:05.338080 4427 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8eafff6-528c-4e61-adff-8aed03dea982-workload-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:38:05.338181 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:05.338097 4427 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8eafff6-528c-4e61-adff-8aed03dea982-istio-data\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:38:05.338181 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:05.338111 4427 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8eafff6-528c-4e61-adff-8aed03dea982-workload-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:38:05.338181 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:05.338125 4427 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8eafff6-528c-4e61-adff-8aed03dea982-credential-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:38:05.338929 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:05.338907 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8eafff6-528c-4e61-adff-8aed03dea982-istio-envoy" pod "f8eafff6-528c-4e61-adff-8aed03dea982" (UID: "f8eafff6-528c-4e61-adff-8aed03dea982"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:38:05.339600 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:05.339559 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8eafff6-528c-4e61-adff-8aed03dea982-istio-podinfo" pod "f8eafff6-528c-4e61-adff-8aed03dea982" (UID: "f8eafff6-528c-4e61-adff-8aed03dea982"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:38:05.340281 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:05.340254 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8eafff6-528c-4e61-adff-8aed03dea982-istio-token" pod "f8eafff6-528c-4e61-adff-8aed03dea982" (UID: "f8eafff6-528c-4e61-adff-8aed03dea982"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:38:05.341430 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:05.341359 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8eafff6-528c-4e61-adff-8aed03dea982-kube-api-access-m86xd" pod "f8eafff6-528c-4e61-adff-8aed03dea982" (UID: "f8eafff6-528c-4e61-adff-8aed03dea982"). InnerVolumeSpecName "kube-api-access-m86xd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:38:05.350435 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:05.350353 4427 generic.go:356] "Generic (PLEG): container finished" podID="f8eafff6-528c-4e61-adff-8aed03dea982" containerID="a4acdb033cd6d67a4f1297a7d368d8477e0472d284b6ecc3984a7adf9d07717d" exitCode=0 Nov 02 01:38:05.350571 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:05.350443 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4" Nov 02 01:38:05.350571 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:05.350449 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4" event={"ID":"f8eafff6-528c-4e61-adff-8aed03dea982","Type":"ContainerDied","Data":"a4acdb033cd6d67a4f1297a7d368d8477e0472d284b6ecc3984a7adf9d07717d"} Nov 02 01:38:05.350571 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:05.350547 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4" event={"ID":"f8eafff6-528c-4e61-adff-8aed03dea982","Type":"ContainerDied","Data":"1c571d9175c498b37f7201d47eee07717c62e2777eb60e7127929e43c93d8b6c"} Nov 02 01:38:05.350734 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:05.350579 4427 scope.go:122] "RemoveContainer" containerID="a4acdb033cd6d67a4f1297a7d368d8477e0472d284b6ecc3984a7adf9d07717d" Nov 02 01:38:05.392737 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:05.392719 4427 scope.go:122] "RemoveContainer" containerID="a4acdb033cd6d67a4f1297a7d368d8477e0472d284b6ecc3984a7adf9d07717d" Nov 02 01:38:05.393138 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:38:05.393112 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4acdb033cd6d67a4f1297a7d368d8477e0472d284b6ecc3984a7adf9d07717d\": container with ID starting with a4acdb033cd6d67a4f1297a7d368d8477e0472d284b6ecc3984a7adf9d07717d not found: ID does not exist" containerID="a4acdb033cd6d67a4f1297a7d368d8477e0472d284b6ecc3984a7adf9d07717d" Nov 02 01:38:05.393207 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:05.393153 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4acdb033cd6d67a4f1297a7d368d8477e0472d284b6ecc3984a7adf9d07717d"} err="failed to get container status \"a4acdb033cd6d67a4f1297a7d368d8477e0472d284b6ecc3984a7adf9d07717d\": rpc error: code = NotFound desc = could not find container \"a4acdb033cd6d67a4f1297a7d368d8477e0472d284b6ecc3984a7adf9d07717d\": container with ID starting with a4acdb033cd6d67a4f1297a7d368d8477e0472d284b6ecc3984a7adf9d07717d not found: ID does not exist" Nov 02 01:38:05.412090 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:05.412054 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4"] Nov 02 01:38:05.416747 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:05.416525 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-50537f4f-openshift-default-58f89bdd97-54nj4"] Nov 02 01:38:05.447458 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:05.442229 4427 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8eafff6-528c-4e61-adff-8aed03dea982-istio-podinfo\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:38:05.447458 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:05.442272 4427 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8eafff6-528c-4e61-adff-8aed03dea982-istio-envoy\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:38:05.447458 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:05.442290 4427 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8eafff6-528c-4e61-adff-8aed03dea982-istio-token\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:38:05.447458 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:05.442304 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m86xd\" (UniqueName: \"kubernetes.io/projected/f8eafff6-528c-4e61-adff-8aed03dea982-kube-api-access-m86xd\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:38:05.539680 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:05.539599 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8eafff6-528c-4e61-adff-8aed03dea982" path="/var/lib/kubelet/pods/f8eafff6-528c-4e61-adff-8aed03dea982/volumes" Nov 02 01:38:06.952762 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:06.952672 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-12e7cb-859c4c76df-xgbzt"] Nov 02 01:38:06.953900 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:06.953872 4427 cpu_manager.go:414] "containerMap: removing container" podUID="f8eafff6-528c-4e61-adff-8aed03dea982" containerName="istio-proxy" Nov 02 01:38:06.953900 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:06.953901 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8eafff6-528c-4e61-adff-8aed03dea982" containerName="istio-proxy" Nov 02 01:38:06.954154 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:06.954052 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8eafff6-528c-4e61-adff-8aed03dea982" containerName="istio-proxy" Nov 02 01:38:06.959895 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:06.959871 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12e7cb-859c4c76df-xgbzt" Nov 02 01:38:06.964590 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:06.964569 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-12e7cb-serving-cert\"" Nov 02 01:38:06.972983 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:06.970430 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-12e7cb-859c4c76df-xgbzt"] Nov 02 01:38:07.074195 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:07.074155 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db04af92-365b-4d73-970a-1da99dc55a3a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-12e7cb-859c4c76df-xgbzt\" (UID: \"db04af92-365b-4d73-970a-1da99dc55a3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12e7cb-859c4c76df-xgbzt" Nov 02 01:38:07.074418 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:07.074320 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db04af92-365b-4d73-970a-1da99dc55a3a-metrics-certs\") pod \"maas-api-e2e-inf-b-12e7cb-859c4c76df-xgbzt\" (UID: \"db04af92-365b-4d73-970a-1da99dc55a3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12e7cb-859c4c76df-xgbzt" Nov 02 01:38:07.074518 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:07.074479 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2msh4\" (UniqueName: \"kubernetes.io/projected/db04af92-365b-4d73-970a-1da99dc55a3a-kube-api-access-2msh4\") pod \"maas-api-e2e-inf-b-12e7cb-859c4c76df-xgbzt\" (UID: \"db04af92-365b-4d73-970a-1da99dc55a3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12e7cb-859c4c76df-xgbzt" Nov 02 01:38:07.074599 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:07.074539 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db04af92-365b-4d73-970a-1da99dc55a3a-maas-api-tls\") pod \"maas-api-e2e-inf-b-12e7cb-859c4c76df-xgbzt\" (UID: \"db04af92-365b-4d73-970a-1da99dc55a3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12e7cb-859c4c76df-xgbzt" Nov 02 01:38:07.175844 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:07.175801 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db04af92-365b-4d73-970a-1da99dc55a3a-metrics-certs\") pod \"maas-api-e2e-inf-b-12e7cb-859c4c76df-xgbzt\" (UID: \"db04af92-365b-4d73-970a-1da99dc55a3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12e7cb-859c4c76df-xgbzt" Nov 02 01:38:07.176030 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:07.175902 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2msh4\" (UniqueName: \"kubernetes.io/projected/db04af92-365b-4d73-970a-1da99dc55a3a-kube-api-access-2msh4\") pod \"maas-api-e2e-inf-b-12e7cb-859c4c76df-xgbzt\" (UID: \"db04af92-365b-4d73-970a-1da99dc55a3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12e7cb-859c4c76df-xgbzt" Nov 02 01:38:07.176030 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:07.175930 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db04af92-365b-4d73-970a-1da99dc55a3a-maas-api-tls\") pod \"maas-api-e2e-inf-b-12e7cb-859c4c76df-xgbzt\" (UID: \"db04af92-365b-4d73-970a-1da99dc55a3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12e7cb-859c4c76df-xgbzt" Nov 02 01:38:07.176120 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:07.176105 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db04af92-365b-4d73-970a-1da99dc55a3a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-12e7cb-859c4c76df-xgbzt\" (UID: \"db04af92-365b-4d73-970a-1da99dc55a3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12e7cb-859c4c76df-xgbzt" Nov 02 01:38:07.176827 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:07.176796 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db04af92-365b-4d73-970a-1da99dc55a3a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-12e7cb-859c4c76df-xgbzt\" (UID: \"db04af92-365b-4d73-970a-1da99dc55a3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12e7cb-859c4c76df-xgbzt" Nov 02 01:38:07.178579 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:07.178557 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db04af92-365b-4d73-970a-1da99dc55a3a-metrics-certs\") pod \"maas-api-e2e-inf-b-12e7cb-859c4c76df-xgbzt\" (UID: \"db04af92-365b-4d73-970a-1da99dc55a3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12e7cb-859c4c76df-xgbzt" Nov 02 01:38:07.178687 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:07.178616 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db04af92-365b-4d73-970a-1da99dc55a3a-maas-api-tls\") pod \"maas-api-e2e-inf-b-12e7cb-859c4c76df-xgbzt\" (UID: \"db04af92-365b-4d73-970a-1da99dc55a3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12e7cb-859c4c76df-xgbzt" Nov 02 01:38:07.196620 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:07.194736 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2msh4\" (UniqueName: \"kubernetes.io/projected/db04af92-365b-4d73-970a-1da99dc55a3a-kube-api-access-2msh4\") pod \"maas-api-e2e-inf-b-12e7cb-859c4c76df-xgbzt\" (UID: \"db04af92-365b-4d73-970a-1da99dc55a3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12e7cb-859c4c76df-xgbzt" Nov 02 01:38:07.295587 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:07.295500 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12e7cb-859c4c76df-xgbzt" Nov 02 01:38:07.619628 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:07.619594 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-12e7cb-859c4c76df-xgbzt"] Nov 02 01:38:07.619959 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:38:07.619933 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb04af92_365b_4d73_970a_1da99dc55a3a.slice/crio-d88e51f29c79a0b6137570f86a17d58c21711fc70969797ea058d1fb03ab2cf9 WatchSource:0}: Error finding container d88e51f29c79a0b6137570f86a17d58c21711fc70969797ea058d1fb03ab2cf9: Status 404 returned error can't find the container with id d88e51f29c79a0b6137570f86a17d58c21711fc70969797ea058d1fb03ab2cf9 Nov 02 01:38:08.385299 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:08.385267 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12e7cb-859c4c76df-xgbzt" event={"ID":"db04af92-365b-4d73-970a-1da99dc55a3a","Type":"ContainerStarted","Data":"7c43e9fbabab169d1ff473af0abd9611f089388556ec452c772de7a408003023"} Nov 02 01:38:08.385789 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:08.385317 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12e7cb-859c4c76df-xgbzt" event={"ID":"db04af92-365b-4d73-970a-1da99dc55a3a","Type":"ContainerStarted","Data":"d88e51f29c79a0b6137570f86a17d58c21711fc70969797ea058d1fb03ab2cf9"} Nov 02 01:38:08.385789 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:08.385613 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12e7cb-859c4c76df-xgbzt" Nov 02 01:38:08.408302 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:08.408243 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12e7cb-859c4c76df-xgbzt" podStartSLOduration=1.807352783 podStartE2EDuration="2.40822794s" totalImagesPullingTime="600.875157ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:38:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:38:08.405721298 +0000 UTC m=+1393.026726506" watchObservedRunningTime="2026-11-02 01:38:08.40822794 +0000 UTC m=+1393.029233151" Nov 02 01:38:12.662589 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:12.662548 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h"] Nov 02 01:38:12.668864 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:12.668830 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h" Nov 02 01:38:12.676171 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:12.676142 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-8b3bae-openshift-default-dockercfg-x8tvb\"" Nov 02 01:38:12.682082 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:12.682049 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h"] Nov 02 01:38:12.692180 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:12.692148 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qndzn\" (UniqueName: \"kubernetes.io/projected/95975531-e9c3-4fcb-9f93-ddab47bb62ba-kube-api-access-qndzn\") pod \"e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h\" (UID: \"95975531-e9c3-4fcb-9f93-ddab47bb62ba\") " pod="openshift-ingress/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h" Nov 02 01:38:12.692483 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:12.692457 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95975531-e9c3-4fcb-9f93-ddab47bb62ba-istio-token\") pod \"e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h\" (UID: \"95975531-e9c3-4fcb-9f93-ddab47bb62ba\") " pod="openshift-ingress/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h" Nov 02 01:38:12.692598 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:12.692574 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95975531-e9c3-4fcb-9f93-ddab47bb62ba-credential-socket\") pod \"e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h\" (UID: \"95975531-e9c3-4fcb-9f93-ddab47bb62ba\") " pod="openshift-ingress/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h" Nov 02 01:38:12.692727 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:12.692710 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95975531-e9c3-4fcb-9f93-ddab47bb62ba-workload-socket\") pod \"e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h\" (UID: \"95975531-e9c3-4fcb-9f93-ddab47bb62ba\") " pod="openshift-ingress/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h" Nov 02 01:38:12.692805 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:12.692791 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95975531-e9c3-4fcb-9f93-ddab47bb62ba-workload-certs\") pod \"e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h\" (UID: \"95975531-e9c3-4fcb-9f93-ddab47bb62ba\") " pod="openshift-ingress/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h" Nov 02 01:38:12.692847 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:12.692819 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95975531-e9c3-4fcb-9f93-ddab47bb62ba-istio-podinfo\") pod \"e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h\" (UID: \"95975531-e9c3-4fcb-9f93-ddab47bb62ba\") " pod="openshift-ingress/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h" Nov 02 01:38:12.692883 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:12.692858 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95975531-e9c3-4fcb-9f93-ddab47bb62ba-istio-data\") pod \"e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h\" (UID: \"95975531-e9c3-4fcb-9f93-ddab47bb62ba\") " pod="openshift-ingress/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h" Nov 02 01:38:12.692930 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:12.692899 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95975531-e9c3-4fcb-9f93-ddab47bb62ba-istio-envoy\") pod \"e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h\" (UID: \"95975531-e9c3-4fcb-9f93-ddab47bb62ba\") " pod="openshift-ingress/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h" Nov 02 01:38:12.692966 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:12.692926 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95975531-e9c3-4fcb-9f93-ddab47bb62ba-istiod-ca-cert\") pod \"e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h\" (UID: \"95975531-e9c3-4fcb-9f93-ddab47bb62ba\") " pod="openshift-ingress/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h" Nov 02 01:38:12.794743 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:12.794701 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95975531-e9c3-4fcb-9f93-ddab47bb62ba-credential-socket\") pod \"e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h\" (UID: \"95975531-e9c3-4fcb-9f93-ddab47bb62ba\") " pod="openshift-ingress/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h" Nov 02 01:38:12.794925 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:12.794868 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95975531-e9c3-4fcb-9f93-ddab47bb62ba-workload-socket\") pod \"e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h\" (UID: \"95975531-e9c3-4fcb-9f93-ddab47bb62ba\") " pod="openshift-ingress/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h" Nov 02 01:38:12.794996 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:12.794944 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95975531-e9c3-4fcb-9f93-ddab47bb62ba-workload-certs\") pod \"e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h\" (UID: \"95975531-e9c3-4fcb-9f93-ddab47bb62ba\") " pod="openshift-ingress/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h" Nov 02 01:38:12.794996 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:12.794969 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95975531-e9c3-4fcb-9f93-ddab47bb62ba-istio-podinfo\") pod \"e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h\" (UID: \"95975531-e9c3-4fcb-9f93-ddab47bb62ba\") " pod="openshift-ingress/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h" Nov 02 01:38:12.795101 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:12.795006 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95975531-e9c3-4fcb-9f93-ddab47bb62ba-istio-data\") pod \"e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h\" (UID: \"95975531-e9c3-4fcb-9f93-ddab47bb62ba\") " pod="openshift-ingress/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h" Nov 02 01:38:12.795101 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:12.795057 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95975531-e9c3-4fcb-9f93-ddab47bb62ba-istio-envoy\") pod \"e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h\" (UID: \"95975531-e9c3-4fcb-9f93-ddab47bb62ba\") " pod="openshift-ingress/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h" Nov 02 01:38:12.795101 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:12.795083 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95975531-e9c3-4fcb-9f93-ddab47bb62ba-istiod-ca-cert\") pod \"e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h\" (UID: \"95975531-e9c3-4fcb-9f93-ddab47bb62ba\") " pod="openshift-ingress/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h" Nov 02 01:38:12.795257 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:12.795121 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95975531-e9c3-4fcb-9f93-ddab47bb62ba-credential-socket\") pod \"e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h\" (UID: \"95975531-e9c3-4fcb-9f93-ddab47bb62ba\") " pod="openshift-ingress/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h" Nov 02 01:38:12.795257 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:12.795202 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95975531-e9c3-4fcb-9f93-ddab47bb62ba-workload-certs\") pod \"e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h\" (UID: \"95975531-e9c3-4fcb-9f93-ddab47bb62ba\") " pod="openshift-ingress/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h" Nov 02 01:38:12.795366 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:12.795279 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95975531-e9c3-4fcb-9f93-ddab47bb62ba-workload-socket\") pod \"e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h\" (UID: \"95975531-e9c3-4fcb-9f93-ddab47bb62ba\") " pod="openshift-ingress/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h" Nov 02 01:38:12.795366 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:12.795317 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qndzn\" (UniqueName: \"kubernetes.io/projected/95975531-e9c3-4fcb-9f93-ddab47bb62ba-kube-api-access-qndzn\") pod \"e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h\" (UID: \"95975531-e9c3-4fcb-9f93-ddab47bb62ba\") " pod="openshift-ingress/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h" Nov 02 01:38:12.795518 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:12.795366 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95975531-e9c3-4fcb-9f93-ddab47bb62ba-istio-token\") pod \"e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h\" (UID: \"95975531-e9c3-4fcb-9f93-ddab47bb62ba\") " pod="openshift-ingress/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h" Nov 02 01:38:12.795518 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:12.795485 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95975531-e9c3-4fcb-9f93-ddab47bb62ba-istio-data\") pod \"e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h\" (UID: \"95975531-e9c3-4fcb-9f93-ddab47bb62ba\") " pod="openshift-ingress/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h" Nov 02 01:38:12.796410 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:12.796353 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95975531-e9c3-4fcb-9f93-ddab47bb62ba-istiod-ca-cert\") pod \"e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h\" (UID: \"95975531-e9c3-4fcb-9f93-ddab47bb62ba\") " pod="openshift-ingress/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h" Nov 02 01:38:12.797804 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:12.797774 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95975531-e9c3-4fcb-9f93-ddab47bb62ba-istio-podinfo\") pod \"e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h\" (UID: \"95975531-e9c3-4fcb-9f93-ddab47bb62ba\") " pod="openshift-ingress/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h" Nov 02 01:38:12.798018 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:12.797994 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95975531-e9c3-4fcb-9f93-ddab47bb62ba-istio-envoy\") pod \"e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h\" (UID: \"95975531-e9c3-4fcb-9f93-ddab47bb62ba\") " pod="openshift-ingress/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h" Nov 02 01:38:12.807714 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:12.807685 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95975531-e9c3-4fcb-9f93-ddab47bb62ba-istio-token\") pod \"e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h\" (UID: \"95975531-e9c3-4fcb-9f93-ddab47bb62ba\") " pod="openshift-ingress/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h" Nov 02 01:38:12.807941 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:12.807918 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qndzn\" (UniqueName: \"kubernetes.io/projected/95975531-e9c3-4fcb-9f93-ddab47bb62ba-kube-api-access-qndzn\") pod \"e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h\" (UID: \"95975531-e9c3-4fcb-9f93-ddab47bb62ba\") " pod="openshift-ingress/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h" Nov 02 01:38:13.003451 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:12.998708 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h" Nov 02 01:38:13.505706 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:13.505658 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h"] Nov 02 01:38:13.618414 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:13.618361 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:38:13.618554 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:13.618502 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:38:13.618620 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:13.618551 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:38:14.400230 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:14.400198 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12e7cb-859c4c76df-xgbzt" Nov 02 01:38:14.625449 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:14.625409 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h" event={"ID":"95975531-e9c3-4fcb-9f93-ddab47bb62ba","Type":"ContainerStarted","Data":"e75bccb167e5e1f6882788fe66536261d715bbce3d95a0d95191a10c531b4863"} Nov 02 01:38:14.625644 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:14.625532 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h" event={"ID":"95975531-e9c3-4fcb-9f93-ddab47bb62ba","Type":"ContainerStarted","Data":"5aae6a4df1b75decabd9210387438af2add7061575bda9af6e20125c069d5b9e"} Nov 02 01:38:14.653986 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:14.653880 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h" podStartSLOduration=2.653863961 podStartE2EDuration="2.653863961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:38:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:38:14.647923999 +0000 UTC m=+1399.268929238" watchObservedRunningTime="2026-11-02 01:38:14.653863961 +0000 UTC m=+1399.274869163" Nov 02 01:38:15.000091 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:15.000000 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h" Nov 02 01:38:15.006948 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:15.006888 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h" Nov 02 01:38:15.634539 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:15.634507 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h" Nov 02 01:38:15.635646 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:15.635628 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h" Nov 02 01:38:17.388394 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:17.388342 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-edef28-59dc7b55f9-vmbqt"] Nov 02 01:38:17.421892 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:17.421855 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-edef28-59dc7b55f9-vmbqt"] Nov 02 01:38:17.422072 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:17.422033 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edef28-59dc7b55f9-vmbqt" Nov 02 01:38:17.439074 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:17.439035 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-edef28-dockercfg-ghrxl\"" Nov 02 01:38:17.439255 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:17.439114 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-edef28\"" Nov 02 01:38:17.515993 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:17.515953 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvcjs\" (UniqueName: \"kubernetes.io/projected/112b8e02-4d50-426c-a6a5-a6d072abfaed-kube-api-access-qvcjs\") pod \"payload-pre-processing-e2e-inf-a-edef28-59dc7b55f9-vmbqt\" (UID: \"112b8e02-4d50-426c-a6a5-a6d072abfaed\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edef28-59dc7b55f9-vmbqt" Nov 02 01:38:17.516486 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:17.516455 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/112b8e02-4d50-426c-a6a5-a6d072abfaed-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-edef28-59dc7b55f9-vmbqt\" (UID: \"112b8e02-4d50-426c-a6a5-a6d072abfaed\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edef28-59dc7b55f9-vmbqt" Nov 02 01:38:17.635401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:17.633280 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvcjs\" (UniqueName: \"kubernetes.io/projected/112b8e02-4d50-426c-a6a5-a6d072abfaed-kube-api-access-qvcjs\") pod \"payload-pre-processing-e2e-inf-a-edef28-59dc7b55f9-vmbqt\" (UID: \"112b8e02-4d50-426c-a6a5-a6d072abfaed\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edef28-59dc7b55f9-vmbqt" Nov 02 01:38:17.635401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:17.633788 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/112b8e02-4d50-426c-a6a5-a6d072abfaed-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-edef28-59dc7b55f9-vmbqt\" (UID: \"112b8e02-4d50-426c-a6a5-a6d072abfaed\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edef28-59dc7b55f9-vmbqt" Nov 02 01:38:17.665830 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:17.663294 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/112b8e02-4d50-426c-a6a5-a6d072abfaed-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-edef28-59dc7b55f9-vmbqt\" (UID: \"112b8e02-4d50-426c-a6a5-a6d072abfaed\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edef28-59dc7b55f9-vmbqt" Nov 02 01:38:17.728458 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:17.728425 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvcjs\" (UniqueName: \"kubernetes.io/projected/112b8e02-4d50-426c-a6a5-a6d072abfaed-kube-api-access-qvcjs\") pod \"payload-pre-processing-e2e-inf-a-edef28-59dc7b55f9-vmbqt\" (UID: \"112b8e02-4d50-426c-a6a5-a6d072abfaed\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edef28-59dc7b55f9-vmbqt" Nov 02 01:38:17.771808 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:17.771774 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edef28-59dc7b55f9-vmbqt" Nov 02 01:38:18.366539 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:18.366506 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-edef28-59dc7b55f9-vmbqt"] Nov 02 01:38:18.368055 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:38:18.368029 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod112b8e02_4d50_426c_a6a5_a6d072abfaed.slice/crio-52de33253a9c11361e37abd7e16aacc012d22cc949d3a1d9fc10b559cfc7033a WatchSource:0}: Error finding container 52de33253a9c11361e37abd7e16aacc012d22cc949d3a1d9fc10b559cfc7033a: Status 404 returned error can't find the container with id 52de33253a9c11361e37abd7e16aacc012d22cc949d3a1d9fc10b559cfc7033a Nov 02 01:38:18.782013 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:18.781919 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edef28-59dc7b55f9-vmbqt" event={"ID":"112b8e02-4d50-426c-a6a5-a6d072abfaed","Type":"ContainerStarted","Data":"419e5518317864f9501a89470d3bbe9ed5d54ecc6d42c605c8ab84063219dca3"} Nov 02 01:38:18.782013 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:18.781968 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edef28-59dc7b55f9-vmbqt" event={"ID":"112b8e02-4d50-426c-a6a5-a6d072abfaed","Type":"ContainerStarted","Data":"52de33253a9c11361e37abd7e16aacc012d22cc949d3a1d9fc10b559cfc7033a"} Nov 02 01:38:18.782663 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:18.782052 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edef28-59dc7b55f9-vmbqt" Nov 02 01:38:18.803264 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:18.803191 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edef28-59dc7b55f9-vmbqt" podStartSLOduration=1.803176128 podStartE2EDuration="1.803176128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:38:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:38:18.800574777 +0000 UTC m=+1403.421579989" watchObservedRunningTime="2026-11-02 01:38:18.803176128 +0000 UTC m=+1403.424181337" Nov 02 01:38:25.286588 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:25.278471 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" podUID="c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:38:29.793620 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:29.793589 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edef28-59dc7b55f9-vmbqt" Nov 02 01:38:31.815397 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:38:31.808771 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:38:38.554699 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:38.554663 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-edef28-76775f884d-lkrgl"] Nov 02 01:38:38.560221 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:38.560196 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edef28-76775f884d-lkrgl" Nov 02 01:38:38.566699 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:38.566188 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-edef28-serving-cert\"" Nov 02 01:38:38.576102 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:38.576073 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-edef28-76775f884d-lkrgl"] Nov 02 01:38:38.726175 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:38.726119 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cdaf63a-0f4f-4329-9a71-e6c0a410fec4-maas-api-tls\") pod \"maas-api-e2e-inf-a-edef28-76775f884d-lkrgl\" (UID: \"2cdaf63a-0f4f-4329-9a71-e6c0a410fec4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edef28-76775f884d-lkrgl" Nov 02 01:38:38.726619 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:38.726592 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ghwq\" (UniqueName: \"kubernetes.io/projected/2cdaf63a-0f4f-4329-9a71-e6c0a410fec4-kube-api-access-4ghwq\") pod \"maas-api-e2e-inf-a-edef28-76775f884d-lkrgl\" (UID: \"2cdaf63a-0f4f-4329-9a71-e6c0a410fec4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edef28-76775f884d-lkrgl" Nov 02 01:38:38.726887 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:38.726871 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cdaf63a-0f4f-4329-9a71-e6c0a410fec4-openshift-service-ca\") pod \"maas-api-e2e-inf-a-edef28-76775f884d-lkrgl\" (UID: \"2cdaf63a-0f4f-4329-9a71-e6c0a410fec4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edef28-76775f884d-lkrgl" Nov 02 01:38:38.727425 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:38.727404 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cdaf63a-0f4f-4329-9a71-e6c0a410fec4-metrics-certs\") pod \"maas-api-e2e-inf-a-edef28-76775f884d-lkrgl\" (UID: \"2cdaf63a-0f4f-4329-9a71-e6c0a410fec4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edef28-76775f884d-lkrgl" Nov 02 01:38:38.834733 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:38.834641 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cdaf63a-0f4f-4329-9a71-e6c0a410fec4-metrics-certs\") pod \"maas-api-e2e-inf-a-edef28-76775f884d-lkrgl\" (UID: \"2cdaf63a-0f4f-4329-9a71-e6c0a410fec4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edef28-76775f884d-lkrgl" Nov 02 01:38:38.835059 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:38.835028 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cdaf63a-0f4f-4329-9a71-e6c0a410fec4-maas-api-tls\") pod \"maas-api-e2e-inf-a-edef28-76775f884d-lkrgl\" (UID: \"2cdaf63a-0f4f-4329-9a71-e6c0a410fec4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edef28-76775f884d-lkrgl" Nov 02 01:38:38.835218 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:38.835201 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ghwq\" (UniqueName: \"kubernetes.io/projected/2cdaf63a-0f4f-4329-9a71-e6c0a410fec4-kube-api-access-4ghwq\") pod \"maas-api-e2e-inf-a-edef28-76775f884d-lkrgl\" (UID: \"2cdaf63a-0f4f-4329-9a71-e6c0a410fec4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edef28-76775f884d-lkrgl" Nov 02 01:38:38.835457 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:38.835440 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cdaf63a-0f4f-4329-9a71-e6c0a410fec4-openshift-service-ca\") pod \"maas-api-e2e-inf-a-edef28-76775f884d-lkrgl\" (UID: \"2cdaf63a-0f4f-4329-9a71-e6c0a410fec4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edef28-76775f884d-lkrgl" Nov 02 01:38:38.838019 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:38.837996 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cdaf63a-0f4f-4329-9a71-e6c0a410fec4-openshift-service-ca\") pod \"maas-api-e2e-inf-a-edef28-76775f884d-lkrgl\" (UID: \"2cdaf63a-0f4f-4329-9a71-e6c0a410fec4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edef28-76775f884d-lkrgl" Nov 02 01:38:38.842491 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:38.842099 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cdaf63a-0f4f-4329-9a71-e6c0a410fec4-metrics-certs\") pod \"maas-api-e2e-inf-a-edef28-76775f884d-lkrgl\" (UID: \"2cdaf63a-0f4f-4329-9a71-e6c0a410fec4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edef28-76775f884d-lkrgl" Nov 02 01:38:38.849653 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:38.849586 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cdaf63a-0f4f-4329-9a71-e6c0a410fec4-maas-api-tls\") pod \"maas-api-e2e-inf-a-edef28-76775f884d-lkrgl\" (UID: \"2cdaf63a-0f4f-4329-9a71-e6c0a410fec4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edef28-76775f884d-lkrgl" Nov 02 01:38:38.862702 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:38.862674 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ghwq\" (UniqueName: \"kubernetes.io/projected/2cdaf63a-0f4f-4329-9a71-e6c0a410fec4-kube-api-access-4ghwq\") pod \"maas-api-e2e-inf-a-edef28-76775f884d-lkrgl\" (UID: \"2cdaf63a-0f4f-4329-9a71-e6c0a410fec4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edef28-76775f884d-lkrgl" Nov 02 01:38:38.879701 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:38.879672 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edef28-76775f884d-lkrgl" Nov 02 01:38:39.186441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:39.179282 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-edef28-76775f884d-lkrgl"] Nov 02 01:38:39.186441 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:38:39.179526 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cdaf63a_0f4f_4329_9a71_e6c0a410fec4.slice/crio-58559291f5946f3df5f72ccc90cc0357f669138771e1fc003f52924ffc77e1dd WatchSource:0}: Error finding container 58559291f5946f3df5f72ccc90cc0357f669138771e1fc003f52924ffc77e1dd: Status 404 returned error can't find the container with id 58559291f5946f3df5f72ccc90cc0357f669138771e1fc003f52924ffc77e1dd Nov 02 01:38:40.124571 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:40.124534 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edef28-76775f884d-lkrgl" event={"ID":"2cdaf63a-0f4f-4329-9a71-e6c0a410fec4","Type":"ContainerStarted","Data":"0ade303333152e7812ef5638eb7aa8e529330fceede9f00c614d64d7e808ff1f"} Nov 02 01:38:40.125014 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:40.124600 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edef28-76775f884d-lkrgl" event={"ID":"2cdaf63a-0f4f-4329-9a71-e6c0a410fec4","Type":"ContainerStarted","Data":"58559291f5946f3df5f72ccc90cc0357f669138771e1fc003f52924ffc77e1dd"} Nov 02 01:38:40.125014 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:40.124709 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edef28-76775f884d-lkrgl" Nov 02 01:38:40.149951 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:40.149855 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edef28-76775f884d-lkrgl" podStartSLOduration=1.636398102 podStartE2EDuration="2.149838961s" totalImagesPullingTime="513.440859ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:38:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:38:40.144826077 +0000 UTC m=+1424.765831288" watchObservedRunningTime="2026-11-02 01:38:40.149838961 +0000 UTC m=+1424.770844172" Nov 02 01:38:46.139770 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:46.139734 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edef28-76775f884d-lkrgl" Nov 02 01:38:48.298695 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:48.298656 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g"] Nov 02 01:38:48.305156 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:48.305133 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g" Nov 02 01:38:48.315119 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:48.315085 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-2589b2-openshift-default-dockercfg-mt8kv\"" Nov 02 01:38:48.316471 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:48.316445 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g"] Nov 02 01:38:48.349571 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:48.349538 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b8221ff-cb53-42c0-a697-4992c40d8e52-workload-certs\") pod \"e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g\" (UID: \"6b8221ff-cb53-42c0-a697-4992c40d8e52\") " pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g" Nov 02 01:38:48.349744 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:48.349587 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrx4s\" (UniqueName: \"kubernetes.io/projected/6b8221ff-cb53-42c0-a697-4992c40d8e52-kube-api-access-jrx4s\") pod \"e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g\" (UID: \"6b8221ff-cb53-42c0-a697-4992c40d8e52\") " pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g" Nov 02 01:38:48.349744 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:48.349673 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b8221ff-cb53-42c0-a697-4992c40d8e52-istio-token\") pod \"e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g\" (UID: \"6b8221ff-cb53-42c0-a697-4992c40d8e52\") " pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g" Nov 02 01:38:48.349827 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:48.349776 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b8221ff-cb53-42c0-a697-4992c40d8e52-istio-podinfo\") pod \"e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g\" (UID: \"6b8221ff-cb53-42c0-a697-4992c40d8e52\") " pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g" Nov 02 01:38:48.349827 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:48.349801 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b8221ff-cb53-42c0-a697-4992c40d8e52-workload-socket\") pod \"e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g\" (UID: \"6b8221ff-cb53-42c0-a697-4992c40d8e52\") " pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g" Nov 02 01:38:48.349970 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:48.349950 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b8221ff-cb53-42c0-a697-4992c40d8e52-credential-socket\") pod \"e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g\" (UID: \"6b8221ff-cb53-42c0-a697-4992c40d8e52\") " pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g" Nov 02 01:38:48.350025 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:48.350013 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b8221ff-cb53-42c0-a697-4992c40d8e52-istio-envoy\") pod \"e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g\" (UID: \"6b8221ff-cb53-42c0-a697-4992c40d8e52\") " pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g" Nov 02 01:38:48.350124 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:48.350113 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b8221ff-cb53-42c0-a697-4992c40d8e52-istio-data\") pod \"e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g\" (UID: \"6b8221ff-cb53-42c0-a697-4992c40d8e52\") " pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g" Nov 02 01:38:48.350166 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:48.350135 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b8221ff-cb53-42c0-a697-4992c40d8e52-istiod-ca-cert\") pod \"e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g\" (UID: \"6b8221ff-cb53-42c0-a697-4992c40d8e52\") " pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g" Nov 02 01:38:48.451939 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:48.451906 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b8221ff-cb53-42c0-a697-4992c40d8e52-istio-podinfo\") pod \"e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g\" (UID: \"6b8221ff-cb53-42c0-a697-4992c40d8e52\") " pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g" Nov 02 01:38:48.451939 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:48.451945 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b8221ff-cb53-42c0-a697-4992c40d8e52-workload-socket\") pod \"e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g\" (UID: \"6b8221ff-cb53-42c0-a697-4992c40d8e52\") " pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g" Nov 02 01:38:48.452191 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:48.452044 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b8221ff-cb53-42c0-a697-4992c40d8e52-credential-socket\") pod \"e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g\" (UID: \"6b8221ff-cb53-42c0-a697-4992c40d8e52\") " pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g" Nov 02 01:38:48.452191 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:48.452092 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b8221ff-cb53-42c0-a697-4992c40d8e52-istio-envoy\") pod \"e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g\" (UID: \"6b8221ff-cb53-42c0-a697-4992c40d8e52\") " pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g" Nov 02 01:38:48.452191 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:48.452179 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b8221ff-cb53-42c0-a697-4992c40d8e52-istio-data\") pod \"e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g\" (UID: \"6b8221ff-cb53-42c0-a697-4992c40d8e52\") " pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g" Nov 02 01:38:48.452351 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:48.452197 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b8221ff-cb53-42c0-a697-4992c40d8e52-istiod-ca-cert\") pod \"e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g\" (UID: \"6b8221ff-cb53-42c0-a697-4992c40d8e52\") " pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g" Nov 02 01:38:48.452351 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:48.452309 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b8221ff-cb53-42c0-a697-4992c40d8e52-workload-certs\") pod \"e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g\" (UID: \"6b8221ff-cb53-42c0-a697-4992c40d8e52\") " pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g" Nov 02 01:38:48.452351 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:48.452347 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrx4s\" (UniqueName: \"kubernetes.io/projected/6b8221ff-cb53-42c0-a697-4992c40d8e52-kube-api-access-jrx4s\") pod \"e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g\" (UID: \"6b8221ff-cb53-42c0-a697-4992c40d8e52\") " pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g" Nov 02 01:38:48.452540 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:48.452454 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b8221ff-cb53-42c0-a697-4992c40d8e52-istio-token\") pod \"e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g\" (UID: \"6b8221ff-cb53-42c0-a697-4992c40d8e52\") " pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g" Nov 02 01:38:48.452540 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:48.452507 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b8221ff-cb53-42c0-a697-4992c40d8e52-workload-socket\") pod \"e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g\" (UID: \"6b8221ff-cb53-42c0-a697-4992c40d8e52\") " pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g" Nov 02 01:38:48.452645 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:48.452555 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b8221ff-cb53-42c0-a697-4992c40d8e52-credential-socket\") pod \"e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g\" (UID: \"6b8221ff-cb53-42c0-a697-4992c40d8e52\") " pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g" Nov 02 01:38:48.452692 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:48.452644 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b8221ff-cb53-42c0-a697-4992c40d8e52-istio-data\") pod \"e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g\" (UID: \"6b8221ff-cb53-42c0-a697-4992c40d8e52\") " pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g" Nov 02 01:38:48.453050 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:48.452931 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b8221ff-cb53-42c0-a697-4992c40d8e52-workload-certs\") pod \"e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g\" (UID: \"6b8221ff-cb53-42c0-a697-4992c40d8e52\") " pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g" Nov 02 01:38:48.453182 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:48.453000 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b8221ff-cb53-42c0-a697-4992c40d8e52-istiod-ca-cert\") pod \"e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g\" (UID: \"6b8221ff-cb53-42c0-a697-4992c40d8e52\") " pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g" Nov 02 01:38:48.454665 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:48.454643 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b8221ff-cb53-42c0-a697-4992c40d8e52-istio-envoy\") pod \"e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g\" (UID: \"6b8221ff-cb53-42c0-a697-4992c40d8e52\") " pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g" Nov 02 01:38:48.454913 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:48.454893 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b8221ff-cb53-42c0-a697-4992c40d8e52-istio-podinfo\") pod \"e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g\" (UID: \"6b8221ff-cb53-42c0-a697-4992c40d8e52\") " pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g" Nov 02 01:38:48.464725 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:48.464699 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b8221ff-cb53-42c0-a697-4992c40d8e52-istio-token\") pod \"e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g\" (UID: \"6b8221ff-cb53-42c0-a697-4992c40d8e52\") " pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g" Nov 02 01:38:48.464842 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:48.464773 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrx4s\" (UniqueName: \"kubernetes.io/projected/6b8221ff-cb53-42c0-a697-4992c40d8e52-kube-api-access-jrx4s\") pod \"e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g\" (UID: \"6b8221ff-cb53-42c0-a697-4992c40d8e52\") " pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g" Nov 02 01:38:48.637514 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:48.637474 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g" Nov 02 01:38:48.932160 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:38:48.932114 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b8221ff_cb53_42c0_a697_4992c40d8e52.slice/crio-7ff643b96a21b245a13c9aeebcc61bfccd50ac0498a83af7281f4725420c482f WatchSource:0}: Error finding container 7ff643b96a21b245a13c9aeebcc61bfccd50ac0498a83af7281f4725420c482f: Status 404 returned error can't find the container with id 7ff643b96a21b245a13c9aeebcc61bfccd50ac0498a83af7281f4725420c482f Nov 02 01:38:48.932832 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:48.932804 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g"] Nov 02 01:38:48.935296 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:48.935256 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:38:48.935456 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:48.935326 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:38:48.935456 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:48.935355 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:38:49.260468 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:49.260353 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g" event={"ID":"6b8221ff-cb53-42c0-a697-4992c40d8e52","Type":"ContainerStarted","Data":"1758ac1393ccd3d867b999af3e71362b1a13d3afb4e9c7a6a6da7c8c16705754"} Nov 02 01:38:49.260468 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:49.260442 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g" event={"ID":"6b8221ff-cb53-42c0-a697-4992c40d8e52","Type":"ContainerStarted","Data":"7ff643b96a21b245a13c9aeebcc61bfccd50ac0498a83af7281f4725420c482f"} Nov 02 01:38:49.285830 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:49.285762 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g" podStartSLOduration=1.285746083 podStartE2EDuration="1.285746083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:38:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:38:49.282880961 +0000 UTC m=+1433.903886177" watchObservedRunningTime="2026-11-02 01:38:49.285746083 +0000 UTC m=+1433.906751293" Nov 02 01:38:49.638406 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:49.638360 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g" Nov 02 01:38:50.640224 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:50.640187 4427 patch_prober.go:29] interesting pod/e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 02 01:38:50.640696 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:50.640247 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g" podUID="6b8221ff-cb53-42c0-a697-4992c40d8e52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 02 01:38:51.641528 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:51.641494 4427 patch_prober.go:29] interesting pod/e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 02 01:38:51.642080 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:51.641561 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g" podUID="6b8221ff-cb53-42c0-a697-4992c40d8e52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 02 01:38:52.638357 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:52.638323 4427 patch_prober.go:29] interesting pod/e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 02 01:38:52.638566 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:52.638416 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g" podUID="6b8221ff-cb53-42c0-a697-4992c40d8e52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 02 01:38:52.896755 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:52.896669 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-12e7cb/test-model-12e7cb-kserve-5b5d7468c9-xkkqs"] Nov 02 01:38:52.902434 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:52.902400 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-12e7cb/test-model-12e7cb-kserve-5b5d7468c9-xkkqs" Nov 02 01:38:52.914697 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:52.914654 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-12e7cb\"/\"kube-root-ca.crt\"" Nov 02 01:38:52.914960 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:52.914943 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-12e7cb\"/\"openshift-service-ca.crt\"" Nov 02 01:38:52.919769 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:52.917315 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-12e7cb\"/\"default-dockercfg-952xl\"" Nov 02 01:38:52.919769 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:52.917406 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-12e7cb\"/\"test-model-12e7cb-kserve-self-signed-certs\"" Nov 02 01:38:52.929870 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:52.929833 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-12e7cb/test-model-12e7cb-kserve-5b5d7468c9-xkkqs"] Nov 02 01:38:52.988791 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:52.988745 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21ee65a5-81e6-4736-91ac-6399ba34947e-model-cache\") pod \"test-model-12e7cb-kserve-5b5d7468c9-xkkqs\" (UID: \"21ee65a5-81e6-4736-91ac-6399ba34947e\") " pod="ai-tenant-e2e-inf-b-12e7cb/test-model-12e7cb-kserve-5b5d7468c9-xkkqs" Nov 02 01:38:52.989000 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:52.988855 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzxpt\" (UniqueName: \"kubernetes.io/projected/21ee65a5-81e6-4736-91ac-6399ba34947e-kube-api-access-vzxpt\") pod \"test-model-12e7cb-kserve-5b5d7468c9-xkkqs\" (UID: \"21ee65a5-81e6-4736-91ac-6399ba34947e\") " pod="ai-tenant-e2e-inf-b-12e7cb/test-model-12e7cb-kserve-5b5d7468c9-xkkqs" Nov 02 01:38:52.989000 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:52.988992 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21ee65a5-81e6-4736-91ac-6399ba34947e-tls-certs\") pod \"test-model-12e7cb-kserve-5b5d7468c9-xkkqs\" (UID: \"21ee65a5-81e6-4736-91ac-6399ba34947e\") " pod="ai-tenant-e2e-inf-b-12e7cb/test-model-12e7cb-kserve-5b5d7468c9-xkkqs" Nov 02 01:38:52.989133 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:52.989047 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21ee65a5-81e6-4736-91ac-6399ba34947e-home\") pod \"test-model-12e7cb-kserve-5b5d7468c9-xkkqs\" (UID: \"21ee65a5-81e6-4736-91ac-6399ba34947e\") " pod="ai-tenant-e2e-inf-b-12e7cb/test-model-12e7cb-kserve-5b5d7468c9-xkkqs" Nov 02 01:38:52.989133 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:52.989085 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21ee65a5-81e6-4736-91ac-6399ba34947e-tmp-dir\") pod \"test-model-12e7cb-kserve-5b5d7468c9-xkkqs\" (UID: \"21ee65a5-81e6-4736-91ac-6399ba34947e\") " pod="ai-tenant-e2e-inf-b-12e7cb/test-model-12e7cb-kserve-5b5d7468c9-xkkqs" Nov 02 01:38:52.989266 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:52.989228 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21ee65a5-81e6-4736-91ac-6399ba34947e-dshm\") pod \"test-model-12e7cb-kserve-5b5d7468c9-xkkqs\" (UID: \"21ee65a5-81e6-4736-91ac-6399ba34947e\") " pod="ai-tenant-e2e-inf-b-12e7cb/test-model-12e7cb-kserve-5b5d7468c9-xkkqs" Nov 02 01:38:53.091680 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:53.091642 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21ee65a5-81e6-4736-91ac-6399ba34947e-tls-certs\") pod \"test-model-12e7cb-kserve-5b5d7468c9-xkkqs\" (UID: \"21ee65a5-81e6-4736-91ac-6399ba34947e\") " pod="ai-tenant-e2e-inf-b-12e7cb/test-model-12e7cb-kserve-5b5d7468c9-xkkqs" Nov 02 01:38:53.109220 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:53.109193 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21ee65a5-81e6-4736-91ac-6399ba34947e-home\") pod \"test-model-12e7cb-kserve-5b5d7468c9-xkkqs\" (UID: \"21ee65a5-81e6-4736-91ac-6399ba34947e\") " pod="ai-tenant-e2e-inf-b-12e7cb/test-model-12e7cb-kserve-5b5d7468c9-xkkqs" Nov 02 01:38:53.109874 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:53.109853 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21ee65a5-81e6-4736-91ac-6399ba34947e-tmp-dir\") pod \"test-model-12e7cb-kserve-5b5d7468c9-xkkqs\" (UID: \"21ee65a5-81e6-4736-91ac-6399ba34947e\") " pod="ai-tenant-e2e-inf-b-12e7cb/test-model-12e7cb-kserve-5b5d7468c9-xkkqs" Nov 02 01:38:53.110501 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:53.110483 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21ee65a5-81e6-4736-91ac-6399ba34947e-dshm\") pod \"test-model-12e7cb-kserve-5b5d7468c9-xkkqs\" (UID: \"21ee65a5-81e6-4736-91ac-6399ba34947e\") " pod="ai-tenant-e2e-inf-b-12e7cb/test-model-12e7cb-kserve-5b5d7468c9-xkkqs" Nov 02 01:38:53.111123 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:53.111102 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21ee65a5-81e6-4736-91ac-6399ba34947e-model-cache\") pod \"test-model-12e7cb-kserve-5b5d7468c9-xkkqs\" (UID: \"21ee65a5-81e6-4736-91ac-6399ba34947e\") " pod="ai-tenant-e2e-inf-b-12e7cb/test-model-12e7cb-kserve-5b5d7468c9-xkkqs" Nov 02 01:38:53.111737 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:53.111716 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzxpt\" (UniqueName: \"kubernetes.io/projected/21ee65a5-81e6-4736-91ac-6399ba34947e-kube-api-access-vzxpt\") pod \"test-model-12e7cb-kserve-5b5d7468c9-xkkqs\" (UID: \"21ee65a5-81e6-4736-91ac-6399ba34947e\") " pod="ai-tenant-e2e-inf-b-12e7cb/test-model-12e7cb-kserve-5b5d7468c9-xkkqs" Nov 02 01:38:53.112464 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:53.109774 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21ee65a5-81e6-4736-91ac-6399ba34947e-home\") pod \"test-model-12e7cb-kserve-5b5d7468c9-xkkqs\" (UID: \"21ee65a5-81e6-4736-91ac-6399ba34947e\") " pod="ai-tenant-e2e-inf-b-12e7cb/test-model-12e7cb-kserve-5b5d7468c9-xkkqs" Nov 02 01:38:53.112661 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:53.109090 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21ee65a5-81e6-4736-91ac-6399ba34947e-tls-certs\") pod \"test-model-12e7cb-kserve-5b5d7468c9-xkkqs\" (UID: \"21ee65a5-81e6-4736-91ac-6399ba34947e\") " pod="ai-tenant-e2e-inf-b-12e7cb/test-model-12e7cb-kserve-5b5d7468c9-xkkqs" Nov 02 01:38:53.112761 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:53.111509 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21ee65a5-81e6-4736-91ac-6399ba34947e-model-cache\") pod \"test-model-12e7cb-kserve-5b5d7468c9-xkkqs\" (UID: \"21ee65a5-81e6-4736-91ac-6399ba34947e\") " pod="ai-tenant-e2e-inf-b-12e7cb/test-model-12e7cb-kserve-5b5d7468c9-xkkqs" Nov 02 01:38:53.112861 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:53.110237 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21ee65a5-81e6-4736-91ac-6399ba34947e-tmp-dir\") pod \"test-model-12e7cb-kserve-5b5d7468c9-xkkqs\" (UID: \"21ee65a5-81e6-4736-91ac-6399ba34947e\") " pod="ai-tenant-e2e-inf-b-12e7cb/test-model-12e7cb-kserve-5b5d7468c9-xkkqs" Nov 02 01:38:53.129240 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:53.129205 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21ee65a5-81e6-4736-91ac-6399ba34947e-dshm\") pod \"test-model-12e7cb-kserve-5b5d7468c9-xkkqs\" (UID: \"21ee65a5-81e6-4736-91ac-6399ba34947e\") " pod="ai-tenant-e2e-inf-b-12e7cb/test-model-12e7cb-kserve-5b5d7468c9-xkkqs" Nov 02 01:38:53.138337 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:53.138306 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzxpt\" (UniqueName: \"kubernetes.io/projected/21ee65a5-81e6-4736-91ac-6399ba34947e-kube-api-access-vzxpt\") pod \"test-model-12e7cb-kserve-5b5d7468c9-xkkqs\" (UID: \"21ee65a5-81e6-4736-91ac-6399ba34947e\") " pod="ai-tenant-e2e-inf-b-12e7cb/test-model-12e7cb-kserve-5b5d7468c9-xkkqs" Nov 02 01:38:53.241860 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:53.241763 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-12e7cb/test-model-12e7cb-kserve-5b5d7468c9-xkkqs" Nov 02 01:38:53.638258 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:53.638213 4427 patch_prober.go:29] interesting pod/e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 02 01:38:53.638482 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:53.638294 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g" podUID="6b8221ff-cb53-42c0-a697-4992c40d8e52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 02 01:38:53.900285 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:38:53.900251 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod21ee65a5_81e6_4736_91ac_6399ba34947e.slice/crio-07ae4c2090b707918781ca3f1dbb8174b215c40bbf6f32670ef099b0b98e2306 WatchSource:0}: Error finding container 07ae4c2090b707918781ca3f1dbb8174b215c40bbf6f32670ef099b0b98e2306: Status 404 returned error can't find the container with id 07ae4c2090b707918781ca3f1dbb8174b215c40bbf6f32670ef099b0b98e2306 Nov 02 01:38:53.905431 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:53.904866 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-12e7cb/test-model-12e7cb-kserve-5b5d7468c9-xkkqs"] Nov 02 01:38:54.418653 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:54.418617 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-12e7cb/test-model-12e7cb-kserve-5b5d7468c9-xkkqs" event={"ID":"21ee65a5-81e6-4736-91ac-6399ba34947e","Type":"ContainerStarted","Data":"9630734ee18562779f539a58dd9f4825d78c0d80937338443fb9fe92cc889f11"} Nov 02 01:38:54.418653 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:54.418660 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-12e7cb/test-model-12e7cb-kserve-5b5d7468c9-xkkqs" event={"ID":"21ee65a5-81e6-4736-91ac-6399ba34947e","Type":"ContainerStarted","Data":"07ae4c2090b707918781ca3f1dbb8174b215c40bbf6f32670ef099b0b98e2306"} Nov 02 01:38:54.441470 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:54.441414 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-12e7cb/test-model-12e7cb-kserve-5b5d7468c9-xkkqs" podStartSLOduration=2.441399765 podStartE2EDuration="2.441399765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:38:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:38:54.439227466 +0000 UTC m=+1439.060232676" watchObservedRunningTime="2026-11-02 01:38:54.441399765 +0000 UTC m=+1439.062404980" Nov 02 01:38:54.642624 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:54.642584 4427 patch_prober.go:29] interesting pod/e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 02 01:38:54.642774 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:54.642651 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g" podUID="6b8221ff-cb53-42c0-a697-4992c40d8e52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 02 01:38:55.242542 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:55.242510 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-12e7cb/test-model-12e7cb-kserve-5b5d7468c9-xkkqs" Nov 02 01:38:55.637967 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:55.637936 4427 patch_prober.go:29] interesting pod/e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 02 01:38:55.638144 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:55.637985 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g" podUID="6b8221ff-cb53-42c0-a697-4992c40d8e52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 02 01:38:56.643029 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:56.642996 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g" Nov 02 01:38:56.643568 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:56.643504 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g" Nov 02 01:38:56.644368 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:38:56.644351 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g" Nov 02 01:39:01.673874 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:39:01.673832 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:39:03.247407 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:03.246791 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-12e7cb/test-model-12e7cb-kserve-5b5d7468c9-xkkqs" Nov 02 01:39:03.417176 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:03.417143 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-12e7cb/test-model-12e7cb-kserve-5b5d7468c9-xkkqs" Nov 02 01:39:03.610430 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:03.610365 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-12e7cb/test-model-12e7cb-kserve-5b5d7468c9-xkkqs" Nov 02 01:39:07.896795 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:07.896747 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w"] Nov 02 01:39:07.904075 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:07.904045 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w" Nov 02 01:39:07.917549 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:07.917502 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-14b63f-openshift-default-dockercfg-kkc76\"" Nov 02 01:39:07.923119 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:07.923090 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w"] Nov 02 01:39:08.049024 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:08.048037 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51776e18-2d31-454f-8c33-d7fbb169c601-istio-podinfo\") pod \"e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w\" (UID: \"51776e18-2d31-454f-8c33-d7fbb169c601\") " pod="openshift-ingress/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w" Nov 02 01:39:08.049024 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:08.048113 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvfcg\" (UniqueName: \"kubernetes.io/projected/51776e18-2d31-454f-8c33-d7fbb169c601-kube-api-access-cvfcg\") pod \"e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w\" (UID: \"51776e18-2d31-454f-8c33-d7fbb169c601\") " pod="openshift-ingress/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w" Nov 02 01:39:08.049024 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:08.048180 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51776e18-2d31-454f-8c33-d7fbb169c601-workload-socket\") pod \"e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w\" (UID: \"51776e18-2d31-454f-8c33-d7fbb169c601\") " pod="openshift-ingress/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w" Nov 02 01:39:08.049024 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:08.048304 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51776e18-2d31-454f-8c33-d7fbb169c601-istio-data\") pod \"e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w\" (UID: \"51776e18-2d31-454f-8c33-d7fbb169c601\") " pod="openshift-ingress/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w" Nov 02 01:39:08.049024 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:08.048337 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51776e18-2d31-454f-8c33-d7fbb169c601-istiod-ca-cert\") pod \"e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w\" (UID: \"51776e18-2d31-454f-8c33-d7fbb169c601\") " pod="openshift-ingress/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w" Nov 02 01:39:08.049024 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:08.048538 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51776e18-2d31-454f-8c33-d7fbb169c601-credential-socket\") pod \"e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w\" (UID: \"51776e18-2d31-454f-8c33-d7fbb169c601\") " pod="openshift-ingress/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w" Nov 02 01:39:08.049024 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:08.048626 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51776e18-2d31-454f-8c33-d7fbb169c601-istio-token\") pod \"e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w\" (UID: \"51776e18-2d31-454f-8c33-d7fbb169c601\") " pod="openshift-ingress/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w" Nov 02 01:39:08.049024 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:08.048698 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51776e18-2d31-454f-8c33-d7fbb169c601-workload-certs\") pod \"e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w\" (UID: \"51776e18-2d31-454f-8c33-d7fbb169c601\") " pod="openshift-ingress/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w" Nov 02 01:39:08.049024 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:08.048724 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51776e18-2d31-454f-8c33-d7fbb169c601-istio-envoy\") pod \"e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w\" (UID: \"51776e18-2d31-454f-8c33-d7fbb169c601\") " pod="openshift-ingress/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w" Nov 02 01:39:08.152764 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:08.152665 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51776e18-2d31-454f-8c33-d7fbb169c601-istio-token\") pod \"e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w\" (UID: \"51776e18-2d31-454f-8c33-d7fbb169c601\") " pod="openshift-ingress/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w" Nov 02 01:39:08.153086 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:08.153044 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51776e18-2d31-454f-8c33-d7fbb169c601-workload-certs\") pod \"e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w\" (UID: \"51776e18-2d31-454f-8c33-d7fbb169c601\") " pod="openshift-ingress/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w" Nov 02 01:39:08.153215 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:08.153199 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51776e18-2d31-454f-8c33-d7fbb169c601-istio-envoy\") pod \"e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w\" (UID: \"51776e18-2d31-454f-8c33-d7fbb169c601\") " pod="openshift-ingress/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w" Nov 02 01:39:08.157867 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:08.153453 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51776e18-2d31-454f-8c33-d7fbb169c601-istio-podinfo\") pod \"e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w\" (UID: \"51776e18-2d31-454f-8c33-d7fbb169c601\") " pod="openshift-ingress/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w" Nov 02 01:39:08.158090 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:08.158067 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvfcg\" (UniqueName: \"kubernetes.io/projected/51776e18-2d31-454f-8c33-d7fbb169c601-kube-api-access-cvfcg\") pod \"e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w\" (UID: \"51776e18-2d31-454f-8c33-d7fbb169c601\") " pod="openshift-ingress/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w" Nov 02 01:39:08.158271 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:08.158253 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51776e18-2d31-454f-8c33-d7fbb169c601-workload-socket\") pod \"e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w\" (UID: \"51776e18-2d31-454f-8c33-d7fbb169c601\") " pod="openshift-ingress/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w" Nov 02 01:39:08.158522 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:08.158501 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51776e18-2d31-454f-8c33-d7fbb169c601-istio-data\") pod \"e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w\" (UID: \"51776e18-2d31-454f-8c33-d7fbb169c601\") " pod="openshift-ingress/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w" Nov 02 01:39:08.158652 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:08.158634 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51776e18-2d31-454f-8c33-d7fbb169c601-istiod-ca-cert\") pod \"e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w\" (UID: \"51776e18-2d31-454f-8c33-d7fbb169c601\") " pod="openshift-ingress/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w" Nov 02 01:39:08.158958 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:08.158936 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51776e18-2d31-454f-8c33-d7fbb169c601-credential-socket\") pod \"e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w\" (UID: \"51776e18-2d31-454f-8c33-d7fbb169c601\") " pod="openshift-ingress/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w" Nov 02 01:39:08.159186 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:08.159163 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51776e18-2d31-454f-8c33-d7fbb169c601-workload-certs\") pod \"e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w\" (UID: \"51776e18-2d31-454f-8c33-d7fbb169c601\") " pod="openshift-ingress/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w" Nov 02 01:39:08.173402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:08.160045 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51776e18-2d31-454f-8c33-d7fbb169c601-istiod-ca-cert\") pod \"e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w\" (UID: \"51776e18-2d31-454f-8c33-d7fbb169c601\") " pod="openshift-ingress/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w" Nov 02 01:39:08.202068 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:08.172036 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51776e18-2d31-454f-8c33-d7fbb169c601-istio-podinfo\") pod \"e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w\" (UID: \"51776e18-2d31-454f-8c33-d7fbb169c601\") " pod="openshift-ingress/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w" Nov 02 01:39:08.202297 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:08.172289 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51776e18-2d31-454f-8c33-d7fbb169c601-credential-socket\") pod \"e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w\" (UID: \"51776e18-2d31-454f-8c33-d7fbb169c601\") " pod="openshift-ingress/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w" Nov 02 01:39:08.202429 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:08.174126 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51776e18-2d31-454f-8c33-d7fbb169c601-workload-socket\") pod \"e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w\" (UID: \"51776e18-2d31-454f-8c33-d7fbb169c601\") " pod="openshift-ingress/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w" Nov 02 01:39:08.202517 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:08.174825 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51776e18-2d31-454f-8c33-d7fbb169c601-istio-data\") pod \"e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w\" (UID: \"51776e18-2d31-454f-8c33-d7fbb169c601\") " pod="openshift-ingress/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w" Nov 02 01:39:08.202612 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:08.188520 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51776e18-2d31-454f-8c33-d7fbb169c601-istio-envoy\") pod \"e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w\" (UID: \"51776e18-2d31-454f-8c33-d7fbb169c601\") " pod="openshift-ingress/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w" Nov 02 01:39:08.202696 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:08.190937 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51776e18-2d31-454f-8c33-d7fbb169c601-istio-token\") pod \"e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w\" (UID: \"51776e18-2d31-454f-8c33-d7fbb169c601\") " pod="openshift-ingress/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w" Nov 02 01:39:08.217442 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:08.212428 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvfcg\" (UniqueName: \"kubernetes.io/projected/51776e18-2d31-454f-8c33-d7fbb169c601-kube-api-access-cvfcg\") pod \"e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w\" (UID: \"51776e18-2d31-454f-8c33-d7fbb169c601\") " pod="openshift-ingress/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w" Nov 02 01:39:08.238251 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:08.238215 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w" Nov 02 01:39:09.767633 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:09.767175 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:39:09.801436 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:09.801394 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:39:09.801734 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:09.801708 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:39:09.974070 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:09.973995 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8"] Nov 02 01:39:10.160675 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:10.160608 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" podUID="fb533b5f-3525-454a-8ce2-ea9246194f28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:10.328346 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:10.321633 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8"] Nov 02 01:39:10.328346 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:10.321778 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8" Nov 02 01:39:10.489530 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:10.474400 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" podUID="7085c0ab-7ee3-4665-abad-2977c25f68ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:10.489530 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:10.486948 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w"] Nov 02 01:39:10.491506 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:10.491448 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" podUID="366c78ee-4aaf-4b7c-84b4-17028538cb1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:10.521428 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:10.514836 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" podUID="ea61971c-5a72-4609-b5a0-58a74b324bb7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:10.521428 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:10.520073 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" podUID="353a8a41-21ee-4470-861c-b53833beb106" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:10.540557 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:10.540214 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" podUID="f67c954a-ce5b-41d9-8d99-7d126bd89d59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:10.553414 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:10.551929 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" podUID="d73ae484-a9b8-4565-a346-0d803a1981ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:10.579578 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:10.579474 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-0bbeda/test-model-0bbeda-kserve-5cb56fbff9-lqp9j" podUID="b45e955d-0d37-44c2-96be-42dc39687913" containerName="main" probeResult="failure" output="Get \"https://10.133.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:10.854208 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:10.854102 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z5b4\" (UniqueName: \"kubernetes.io/projected/a6fe33af-426d-4155-98a1-a6dfdaeead32-kube-api-access-9z5b4\") pod \"e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8\" (UID: \"a6fe33af-426d-4155-98a1-a6dfdaeead32\") " pod="openshift-ingress/e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8" Nov 02 01:39:10.854845 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:10.854816 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6fe33af-426d-4155-98a1-a6dfdaeead32-istiod-ca-cert\") pod \"e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8\" (UID: \"a6fe33af-426d-4155-98a1-a6dfdaeead32\") " pod="openshift-ingress/e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8" Nov 02 01:39:10.855059 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:10.855028 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6fe33af-426d-4155-98a1-a6dfdaeead32-workload-socket\") pod \"e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8\" (UID: \"a6fe33af-426d-4155-98a1-a6dfdaeead32\") " pod="openshift-ingress/e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8" Nov 02 01:39:10.877401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:10.873509 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6fe33af-426d-4155-98a1-a6dfdaeead32-workload-certs\") pod \"e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8\" (UID: \"a6fe33af-426d-4155-98a1-a6dfdaeead32\") " pod="openshift-ingress/e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8" Nov 02 01:39:10.877401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:10.873787 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6fe33af-426d-4155-98a1-a6dfdaeead32-istio-envoy\") pod \"e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8\" (UID: \"a6fe33af-426d-4155-98a1-a6dfdaeead32\") " pod="openshift-ingress/e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8" Nov 02 01:39:10.877401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:10.873849 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6fe33af-426d-4155-98a1-a6dfdaeead32-istio-data\") pod \"e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8\" (UID: \"a6fe33af-426d-4155-98a1-a6dfdaeead32\") " pod="openshift-ingress/e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8" Nov 02 01:39:10.877401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:10.873927 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6fe33af-426d-4155-98a1-a6dfdaeead32-istio-podinfo\") pod \"e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8\" (UID: \"a6fe33af-426d-4155-98a1-a6dfdaeead32\") " pod="openshift-ingress/e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8" Nov 02 01:39:10.877401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:10.874007 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6fe33af-426d-4155-98a1-a6dfdaeead32-istio-token\") pod \"e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8\" (UID: \"a6fe33af-426d-4155-98a1-a6dfdaeead32\") " pod="openshift-ingress/e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8" Nov 02 01:39:10.877401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:10.874088 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6fe33af-426d-4155-98a1-a6dfdaeead32-credential-socket\") pod \"e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8\" (UID: \"a6fe33af-426d-4155-98a1-a6dfdaeead32\") " pod="openshift-ingress/e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8" Nov 02 01:39:10.894683 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:10.887616 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-12e7cb/test-model-12e7cb-kserve-5b5d7468c9-xkkqs" podUID="21ee65a5-81e6-4736-91ac-6399ba34947e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.131:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:10.996236 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:10.914811 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" podUID="d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Nov 02 01:39:10.996236 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:10.915132 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" podUID="c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:10.996236 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:10.959357 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-e412d5-openshift-default-dockercfg-6gqnm\"" Nov 02 01:39:10.996236 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:10.959495 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" podUID="cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:10.996236 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:10.959857 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" podUID="706df1bd-19cf-4ab9-9aaa-3c12c97ccf45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:10.996236 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:10.976353 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" podUID="9c056798-2cf8-47b1-9829-67cebda7a248" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:11.111508 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:11.075901 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6fe33af-426d-4155-98a1-a6dfdaeead32-istio-envoy\") pod \"e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8\" (UID: \"a6fe33af-426d-4155-98a1-a6dfdaeead32\") " pod="openshift-ingress/e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8" Nov 02 01:39:11.111508 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:11.075975 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6fe33af-426d-4155-98a1-a6dfdaeead32-istio-data\") pod \"e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8\" (UID: \"a6fe33af-426d-4155-98a1-a6dfdaeead32\") " pod="openshift-ingress/e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8" Nov 02 01:39:11.111508 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:11.076044 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6fe33af-426d-4155-98a1-a6dfdaeead32-istio-podinfo\") pod \"e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8\" (UID: \"a6fe33af-426d-4155-98a1-a6dfdaeead32\") " pod="openshift-ingress/e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8" Nov 02 01:39:11.111508 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:11.076114 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6fe33af-426d-4155-98a1-a6dfdaeead32-istio-token\") pod \"e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8\" (UID: \"a6fe33af-426d-4155-98a1-a6dfdaeead32\") " pod="openshift-ingress/e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8" Nov 02 01:39:11.111508 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:11.076185 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6fe33af-426d-4155-98a1-a6dfdaeead32-credential-socket\") pod \"e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8\" (UID: \"a6fe33af-426d-4155-98a1-a6dfdaeead32\") " pod="openshift-ingress/e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8" Nov 02 01:39:11.111508 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:11.076266 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z5b4\" (UniqueName: \"kubernetes.io/projected/a6fe33af-426d-4155-98a1-a6dfdaeead32-kube-api-access-9z5b4\") pod \"e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8\" (UID: \"a6fe33af-426d-4155-98a1-a6dfdaeead32\") " pod="openshift-ingress/e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8" Nov 02 01:39:11.111508 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:11.076302 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6fe33af-426d-4155-98a1-a6dfdaeead32-istiod-ca-cert\") pod \"e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8\" (UID: \"a6fe33af-426d-4155-98a1-a6dfdaeead32\") " pod="openshift-ingress/e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8" Nov 02 01:39:11.111508 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:11.076393 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6fe33af-426d-4155-98a1-a6dfdaeead32-workload-socket\") pod \"e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8\" (UID: \"a6fe33af-426d-4155-98a1-a6dfdaeead32\") " pod="openshift-ingress/e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8" Nov 02 01:39:11.111508 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:11.076513 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6fe33af-426d-4155-98a1-a6dfdaeead32-workload-certs\") pod \"e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8\" (UID: \"a6fe33af-426d-4155-98a1-a6dfdaeead32\") " pod="openshift-ingress/e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8" Nov 02 01:39:11.111508 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:11.111223 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6fe33af-426d-4155-98a1-a6dfdaeead32-workload-socket\") pod \"e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8\" (UID: \"a6fe33af-426d-4155-98a1-a6dfdaeead32\") " pod="openshift-ingress/e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8" Nov 02 01:39:11.119648 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:11.112775 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6fe33af-426d-4155-98a1-a6dfdaeead32-workload-certs\") pod \"e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8\" (UID: \"a6fe33af-426d-4155-98a1-a6dfdaeead32\") " pod="openshift-ingress/e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8" Nov 02 01:39:11.119648 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:11.113036 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6fe33af-426d-4155-98a1-a6dfdaeead32-credential-socket\") pod \"e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8\" (UID: \"a6fe33af-426d-4155-98a1-a6dfdaeead32\") " pod="openshift-ingress/e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8" Nov 02 01:39:11.119648 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:11.114002 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" podUID="eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:11.119648 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:11.118260 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" podUID="47a9069f-1ffa-4c83-be8b-5e62f1d3275a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 02 01:39:11.129410 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:11.120414 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6fe33af-426d-4155-98a1-a6dfdaeead32-istio-data\") pod \"e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8\" (UID: \"a6fe33af-426d-4155-98a1-a6dfdaeead32\") " pod="openshift-ingress/e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8" Nov 02 01:39:11.167565 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:11.130316 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6fe33af-426d-4155-98a1-a6dfdaeead32-istiod-ca-cert\") pod \"e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8\" (UID: \"a6fe33af-426d-4155-98a1-a6dfdaeead32\") " pod="openshift-ingress/e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8" Nov 02 01:39:11.167565 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:11.143756 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w" event={"ID":"51776e18-2d31-454f-8c33-d7fbb169c601","Type":"ContainerStarted","Data":"c6fa163efa9f23a67ed4f6f8384d79e81973c1f93fee1da481c0db9fe99d998d"} Nov 02 01:39:11.170651 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:11.170607 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" podUID="fb533b5f-3525-454a-8ce2-ea9246194f28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:11.171328 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:11.171301 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" Nov 02 01:39:11.175462 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:11.175394 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6fe33af-426d-4155-98a1-a6dfdaeead32-istio-podinfo\") pod \"e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8\" (UID: \"a6fe33af-426d-4155-98a1-a6dfdaeead32\") " pod="openshift-ingress/e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8" Nov 02 01:39:11.285624 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:11.271613 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6fe33af-426d-4155-98a1-a6dfdaeead32-istio-token\") pod \"e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8\" (UID: \"a6fe33af-426d-4155-98a1-a6dfdaeead32\") " pod="openshift-ingress/e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8" Nov 02 01:39:11.412285 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:11.340562 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z5b4\" (UniqueName: \"kubernetes.io/projected/a6fe33af-426d-4155-98a1-a6dfdaeead32-kube-api-access-9z5b4\") pod \"e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8\" (UID: \"a6fe33af-426d-4155-98a1-a6dfdaeead32\") " pod="openshift-ingress/e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8" Nov 02 01:39:11.412285 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:11.399762 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6fe33af-426d-4155-98a1-a6dfdaeead32-istio-envoy\") pod \"e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8\" (UID: \"a6fe33af-426d-4155-98a1-a6dfdaeead32\") " pod="openshift-ingress/e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8" Nov 02 01:39:11.482602 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:11.477099 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8" Nov 02 01:39:11.482938 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:11.482899 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" podUID="7085c0ab-7ee3-4665-abad-2977c25f68ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:11.483875 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:11.483851 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" Nov 02 01:39:11.565998 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:11.524291 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" podUID="ea61971c-5a72-4609-b5a0-58a74b324bb7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 02 01:39:11.565998 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:11.527473 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" podUID="353a8a41-21ee-4470-861c-b53833beb106" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:11.798406 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:11.772085 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" podUID="f67c954a-ce5b-41d9-8d99-7d126bd89d59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:11.861399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:11.855542 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" Nov 02 01:39:11.861399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:11.855618 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" Nov 02 01:39:12.028250 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:12.028143 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-0bbeda/test-model-0bbeda-kserve-5cb56fbff9-lqp9j" podUID="b45e955d-0d37-44c2-96be-42dc39687913" containerName="main" probeResult="failure" output="Get \"https://10.133.0.117:8000/ready\": context deadline exceeded" Nov 02 01:39:12.152881 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:12.152832 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" podUID="9c056798-2cf8-47b1-9829-67cebda7a248" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:12.209130 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:12.187552 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" podUID="706df1bd-19cf-4ab9-9aaa-3c12c97ccf45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 02 01:39:12.209130 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:12.191219 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" podUID="fb533b5f-3525-454a-8ce2-ea9246194f28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:12.209547 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:12.209512 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" podUID="47a9069f-1ffa-4c83-be8b-5e62f1d3275a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:12.211905 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:12.211873 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" podUID="eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Nov 02 01:39:12.269403 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:12.225694 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-12e7cb/test-model-12e7cb-kserve-5b5d7468c9-xkkqs" podUID="21ee65a5-81e6-4736-91ac-6399ba34947e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.131:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:12.347459 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:12.344955 4427 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" podUID="d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": context deadline exceeded" Nov 02 01:39:12.357165 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:12.356484 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" podUID="c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:12.429560 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:12.423905 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" podUID="366c78ee-4aaf-4b7c-84b4-17028538cb1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:12.479401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:12.474225 4427 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" podUID="d73ae484-a9b8-4565-a346-0d803a1981ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": context deadline exceeded" Nov 02 01:39:12.519446 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:12.484227 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" podUID="7085c0ab-7ee3-4665-abad-2977c25f68ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Nov 02 01:39:12.531703 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:12.531662 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" podUID="ea61971c-5a72-4609-b5a0-58a74b324bb7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:12.544314 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:12.544272 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" podUID="353a8a41-21ee-4470-861c-b53833beb106" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:12.823417 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:12.819536 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" podUID="d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:12.823417 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:12.819858 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" podUID="f67c954a-ce5b-41d9-8d99-7d126bd89d59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:13.060404 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:13.029532 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-0bbeda/test-model-0bbeda-kserve-5cb56fbff9-lqp9j" podUID="b45e955d-0d37-44c2-96be-42dc39687913" containerName="main" probeResult="failure" output="Get \"https://10.133.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:13.166405 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:13.153950 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" podUID="9c056798-2cf8-47b1-9829-67cebda7a248" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 01:39:13.202413 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:13.187824 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" podUID="706df1bd-19cf-4ab9-9aaa-3c12c97ccf45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:13.202413 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:13.191531 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66" podUID="d73ae484-a9b8-4565-a346-0d803a1981ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:13.202413 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:13.191789 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" podUID="fb533b5f-3525-454a-8ce2-ea9246194f28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Nov 02 01:39:13.212521 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:39:13.212488 4427 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.357s" Nov 02 01:39:13.212944 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:13.212924 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" Nov 02 01:39:13.213156 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:13.213127 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-0bbeda/test-model-0bbeda-kserve-5cb56fbff9-lqp9j" Nov 02 01:39:13.214442 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:13.213489 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" podUID="eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:13.214805 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:13.213539 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" podUID="47a9069f-1ffa-4c83-be8b-5e62f1d3275a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 02 01:39:13.242466 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:13.238102 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-12e7cb/test-model-12e7cb-kserve-5b5d7468c9-xkkqs" podUID="21ee65a5-81e6-4736-91ac-6399ba34947e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.131:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:13.267405 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:13.244011 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" Nov 02 01:39:13.267405 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:13.244078 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" Nov 02 01:39:13.267405 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:13.244116 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" Nov 02 01:39:13.267405 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:13.244148 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" Nov 02 01:39:13.267405 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:13.244180 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-12e7cb/test-model-12e7cb-kserve-5b5d7468c9-xkkqs" Nov 02 01:39:13.267405 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:13.244214 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" Nov 02 01:39:13.365859 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:13.358588 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" podUID="c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:13.450784 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:13.436966 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" podUID="366c78ee-4aaf-4b7c-84b4-17028538cb1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 02 01:39:13.537043 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:13.536952 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" Nov 02 01:39:13.557450 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:13.557396 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" podUID="7085c0ab-7ee3-4665-abad-2977c25f68ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Nov 02 01:39:13.805457 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:13.804364 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" podUID="ea61971c-5a72-4609-b5a0-58a74b324bb7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 02 01:39:13.805457 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:13.804737 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" podUID="353a8a41-21ee-4470-861c-b53833beb106" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 02 01:39:13.835416 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:13.832881 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" podUID="f67c954a-ce5b-41d9-8d99-7d126bd89d59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 02 01:39:13.835416 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:13.833708 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" podUID="cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:13.850444 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:13.848586 4427 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" podUID="ea61971c-5a72-4609-b5a0-58a74b324bb7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": context deadline exceeded" Nov 02 01:39:13.882400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:13.877558 4427 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-0bbeda/test-model-0bbeda-kserve-5cb56fbff9-lqp9j" podUID="b45e955d-0d37-44c2-96be-42dc39687913" containerName="main" probeResult="failure" output="Get \"https://10.133.0.117:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:13.882575 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:13.882526 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" podUID="d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:13.950522 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:13.949513 4427 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" podUID="706df1bd-19cf-4ab9-9aaa-3c12c97ccf45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": context deadline exceeded" Nov 02 01:39:14.085230 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:14.085076 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-0bbeda/test-model-0bbeda-kserve-5cb56fbff9-lqp9j" podUID="b45e955d-0d37-44c2-96be-42dc39687913" containerName="main" probeResult="failure" output="Get \"https://10.133.0.117:8000/ready\": context deadline exceeded" Nov 02 01:39:14.252422 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:14.205507 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" podUID="fb533b5f-3525-454a-8ce2-ea9246194f28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:14.252422 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:14.207294 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" podUID="9c056798-2cf8-47b1-9829-67cebda7a248" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:14.274422 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:14.259536 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" podUID="47a9069f-1ffa-4c83-be8b-5e62f1d3275a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:14.287246 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:14.277248 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-12e7cb/test-model-12e7cb-kserve-5b5d7468c9-xkkqs" podUID="21ee65a5-81e6-4736-91ac-6399ba34947e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.131:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:14.289808 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:14.278071 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" podUID="706df1bd-19cf-4ab9-9aaa-3c12c97ccf45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:14.564432 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:14.564083 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" podUID="366c78ee-4aaf-4b7c-84b4-17028538cb1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 02 01:39:14.581301 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:14.580698 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" podUID="7085c0ab-7ee3-4665-abad-2977c25f68ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:14.581506 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:14.581446 4427 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-12e7cb/test-model-12e7cb-kserve-5b5d7468c9-xkkqs" podUID="21ee65a5-81e6-4736-91ac-6399ba34947e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.131:8000/health\": context deadline exceeded" Nov 02 01:39:14.844139 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:14.844046 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" podUID="353a8a41-21ee-4470-861c-b53833beb106" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:14.871606 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:14.871561 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" podUID="ea61971c-5a72-4609-b5a0-58a74b324bb7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:14.913715 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:14.913666 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" podUID="d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Nov 02 01:39:15.127837 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:15.127789 4427 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" podUID="7085c0ab-7ee3-4665-abad-2977c25f68ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:15.140890 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:15.140850 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" podUID="cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 02 01:39:15.148074 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:15.148028 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:39:15.154687 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:15.154644 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:39:15.262459 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:39:15.232891 4427 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.433s" Nov 02 01:39:15.356431 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:15.305867 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" Nov 02 01:39:15.356431 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:15.305933 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" Nov 02 01:39:15.356431 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:15.306037 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" Nov 02 01:39:15.356431 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:15.306070 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-0bbeda/test-model-0bbeda-kserve-5cb56fbff9-lqp9j" Nov 02 01:39:15.356431 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:15.306099 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" Nov 02 01:39:15.356431 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:15.306134 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" Nov 02 01:39:15.356431 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:15.306164 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" Nov 02 01:39:15.356431 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:15.306201 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" Nov 02 01:39:15.356431 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:15.306240 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" Nov 02 01:39:15.356431 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:15.306275 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" Nov 02 01:39:15.356431 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:15.306327 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" Nov 02 01:39:15.356431 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:15.263751 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:39:15.356431 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:15.291638 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" podUID="706df1bd-19cf-4ab9-9aaa-3c12c97ccf45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:15.356431 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:15.291690 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-12e7cb/test-model-12e7cb-kserve-5b5d7468c9-xkkqs" podUID="21ee65a5-81e6-4736-91ac-6399ba34947e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.131:8000/ready\": context deadline exceeded" Nov 02 01:39:15.409473 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:15.373634 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" Nov 02 01:39:15.681743 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:15.661424 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" podUID="7085c0ab-7ee3-4665-abad-2977c25f68ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Nov 02 01:39:15.889991 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:15.872535 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" podUID="ea61971c-5a72-4609-b5a0-58a74b324bb7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:15.935640 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:15.934843 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" podUID="d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:16.145735 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:16.145680 4427 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" podUID="366c78ee-4aaf-4b7c-84b4-17028538cb1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:16.231014 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:16.230895 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" podUID="fb533b5f-3525-454a-8ce2-ea9246194f28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Nov 02 01:39:16.241718 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:16.241644 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" Nov 02 01:39:16.241968 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:16.241944 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w" event={"ID":"51776e18-2d31-454f-8c33-d7fbb169c601","Type":"ContainerStarted","Data":"40531b877859708237bf174c5ce262ff352f62c3d4a21cbd22017be0ad9db7ca"} Nov 02 01:39:16.242106 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:16.242090 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8"] Nov 02 01:39:16.268057 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:16.253547 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w" Nov 02 01:39:16.268057 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:16.267473 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" podUID="366c78ee-4aaf-4b7c-84b4-17028538cb1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:16.303764 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:16.303733 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w" Nov 02 01:39:16.313018 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:16.312979 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-12e7cb/test-model-12e7cb-kserve-5b5d7468c9-xkkqs" podUID="21ee65a5-81e6-4736-91ac-6399ba34947e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.131:8000/ready\": context deadline exceeded" Nov 02 01:39:16.388858 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:16.374591 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" podUID="706df1bd-19cf-4ab9-9aaa-3c12c97ccf45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:16.640240 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:16.640199 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-0bbeda/test-model-0bbeda-kserve-5cb56fbff9-lqp9j" podUID="b45e955d-0d37-44c2-96be-42dc39687913" containerName="main" probeResult="failure" output="Get \"https://10.133.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:16.732219 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:16.732178 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" podUID="7085c0ab-7ee3-4665-abad-2977c25f68ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:16.910624 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:16.910547 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" podUID="ea61971c-5a72-4609-b5a0-58a74b324bb7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:16.912711 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:16.912677 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" podUID="cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 02 01:39:17.280498 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:17.275170 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" Nov 02 01:39:17.330401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:17.322505 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" podUID="366c78ee-4aaf-4b7c-84b4-17028538cb1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:17.330401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:17.322846 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" Nov 02 01:39:17.369347 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:17.361096 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" podUID="47a9069f-1ffa-4c83-be8b-5e62f1d3275a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:17.371247 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:17.370667 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" podUID="fb533b5f-3525-454a-8ce2-ea9246194f28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:17.372392 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:17.371504 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" Nov 02 01:39:17.423328 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:17.423269 4427 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" podUID="cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": context deadline exceeded" Nov 02 01:39:17.569753 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:17.569656 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-12e7cb/test-model-12e7cb-kserve-5b5d7468c9-xkkqs" podUID="21ee65a5-81e6-4736-91ac-6399ba34947e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.131:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:17.626652 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:17.626608 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" podUID="9c056798-2cf8-47b1-9829-67cebda7a248" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:17.690458 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:17.688800 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-0bbeda/test-model-0bbeda-kserve-5cb56fbff9-lqp9j" podUID="b45e955d-0d37-44c2-96be-42dc39687913" containerName="main" probeResult="failure" output="Get \"https://10.133.0.117:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:17.748699 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:17.748653 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" podUID="7085c0ab-7ee3-4665-abad-2977c25f68ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:17.902965 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:17.902836 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-0bbeda/test-model-0bbeda-kserve-5cb56fbff9-lqp9j" Nov 02 01:39:17.929226 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:17.929185 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" podUID="f67c954a-ce5b-41d9-8d99-7d126bd89d59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:17.941075 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:17.940919 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" podUID="d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Nov 02 01:39:17.952531 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:17.952494 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" podUID="ea61971c-5a72-4609-b5a0-58a74b324bb7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:17.960409 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:17.960028 4427 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" podUID="9c056798-2cf8-47b1-9829-67cebda7a248" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:17.960409 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:17.960103 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8" event={"ID":"a6fe33af-426d-4155-98a1-a6dfdaeead32","Type":"ContainerStarted","Data":"a0db62a8741afd6232db998ff821a91006a2990a18326b967f8a037f4bd7e8d4"} Nov 02 01:39:17.960409 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:17.960145 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8" event={"ID":"a6fe33af-426d-4155-98a1-a6dfdaeead32","Type":"ContainerStarted","Data":"9a6a59f6e21034b6bd87f9eb34e9437a3baffa29e9158c2886aba587aae25315"} Nov 02 01:39:17.960409 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:17.960168 4427 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 01:39:17.962515 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:17.962491 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w" Nov 02 01:39:17.967308 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:17.967274 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w" Nov 02 01:39:17.981395 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:17.971959 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w" podStartSLOduration=10.95897365 podStartE2EDuration="10.95897365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:39:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:39:16.546533605 +0000 UTC m=+1461.167538902" watchObservedRunningTime="2026-11-02 01:39:17.95897365 +0000 UTC m=+1462.579978862" Nov 02 01:39:17.986453 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:17.982495 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" podUID="353a8a41-21ee-4470-861c-b53833beb106" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:18.014155 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:18.014109 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" podUID="eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:18.138656 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:18.112589 4427 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" podUID="fb533b5f-3525-454a-8ce2-ea9246194f28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:18.323894 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:18.323837 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" podUID="366c78ee-4aaf-4b7c-84b4-17028538cb1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:18.385399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:18.384981 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" podUID="47a9069f-1ffa-4c83-be8b-5e62f1d3275a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:18.397586 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:18.397529 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" podUID="fb533b5f-3525-454a-8ce2-ea9246194f28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:18.399037 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:18.399003 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" podUID="706df1bd-19cf-4ab9-9aaa-3c12c97ccf45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 02 01:39:18.401085 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:18.401064 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" Nov 02 01:39:18.470550 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:18.459599 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" Nov 02 01:39:18.483164 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:18.472205 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-12e7cb/test-model-12e7cb-kserve-5b5d7468c9-xkkqs" Nov 02 01:39:18.483164 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:18.482002 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8" Nov 02 01:39:18.509939 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:18.505737 4427 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" podUID="c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:39:18.509939 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:18.507659 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" Nov 02 01:39:18.519478 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:18.510532 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8" Nov 02 01:39:18.519478 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:18.511475 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" Nov 02 01:39:18.521310 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:18.521281 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" podUID="c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Nov 02 01:39:18.611873 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:18.611792 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8" podStartSLOduration=9.611768634 podStartE2EDuration="9.611768634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:39:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:39:17.995872433 +0000 UTC m=+1462.616877646" watchObservedRunningTime="2026-11-02 01:39:18.611768634 +0000 UTC m=+1463.232773849" Nov 02 01:39:18.617577 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:18.617533 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" Nov 02 01:39:18.643632 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:18.643598 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-0bbeda/test-model-0bbeda-kserve-5cb56fbff9-lqp9j" Nov 02 01:39:18.644160 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:18.644137 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" Nov 02 01:39:19.010454 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:19.010423 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8" Nov 02 01:39:19.012105 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:19.011466 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8" Nov 02 01:39:22.002615 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:22.002578 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e412d5-5ff4cd49ff-8fb8p"] Nov 02 01:39:22.020719 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:22.020682 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e412d5-5ff4cd49ff-8fb8p" Nov 02 01:39:22.021537 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:22.021508 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e412d5-5ff4cd49ff-8fb8p"] Nov 02 01:39:22.026633 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:22.026451 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-e412d5-dockercfg-g84v6\"" Nov 02 01:39:22.026633 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:22.026523 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-e412d5\"" Nov 02 01:39:22.058470 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:22.058439 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e412d5-68746dfdfc-lrhnf"] Nov 02 01:39:22.070456 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:22.070426 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e412d5-68746dfdfc-lrhnf" Nov 02 01:39:22.075396 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:22.075349 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e412d5-68746dfdfc-lrhnf"] Nov 02 01:39:22.245943 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:22.239879 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s6kj\" (UniqueName: \"kubernetes.io/projected/8f3cd62e-2f33-481d-a827-63d2fddf6a46-kube-api-access-9s6kj\") pod \"payload-processing-e2e-inf-b-e412d5-68746dfdfc-lrhnf\" (UID: \"8f3cd62e-2f33-481d-a827-63d2fddf6a46\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e412d5-68746dfdfc-lrhnf" Nov 02 01:39:22.245943 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:22.240162 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ff70346-02a9-4a43-9b5f-19a83d64d169-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e412d5-5ff4cd49ff-8fb8p\" (UID: \"5ff70346-02a9-4a43-9b5f-19a83d64d169\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e412d5-5ff4cd49ff-8fb8p" Nov 02 01:39:22.245943 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:22.240319 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f3cd62e-2f33-481d-a827-63d2fddf6a46-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e412d5-68746dfdfc-lrhnf\" (UID: \"8f3cd62e-2f33-481d-a827-63d2fddf6a46\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e412d5-68746dfdfc-lrhnf" Nov 02 01:39:22.245943 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:22.240423 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j96j\" (UniqueName: \"kubernetes.io/projected/5ff70346-02a9-4a43-9b5f-19a83d64d169-kube-api-access-2j96j\") pod \"payload-pre-processing-e2e-inf-b-e412d5-5ff4cd49ff-8fb8p\" (UID: \"5ff70346-02a9-4a43-9b5f-19a83d64d169\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e412d5-5ff4cd49ff-8fb8p" Nov 02 01:39:22.355449 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:22.342117 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s6kj\" (UniqueName: \"kubernetes.io/projected/8f3cd62e-2f33-481d-a827-63d2fddf6a46-kube-api-access-9s6kj\") pod \"payload-processing-e2e-inf-b-e412d5-68746dfdfc-lrhnf\" (UID: \"8f3cd62e-2f33-481d-a827-63d2fddf6a46\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e412d5-68746dfdfc-lrhnf" Nov 02 01:39:22.355449 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:22.342338 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ff70346-02a9-4a43-9b5f-19a83d64d169-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e412d5-5ff4cd49ff-8fb8p\" (UID: \"5ff70346-02a9-4a43-9b5f-19a83d64d169\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e412d5-5ff4cd49ff-8fb8p" Nov 02 01:39:22.355449 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:22.342496 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f3cd62e-2f33-481d-a827-63d2fddf6a46-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e412d5-68746dfdfc-lrhnf\" (UID: \"8f3cd62e-2f33-481d-a827-63d2fddf6a46\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e412d5-68746dfdfc-lrhnf" Nov 02 01:39:22.355449 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:22.342572 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j96j\" (UniqueName: \"kubernetes.io/projected/5ff70346-02a9-4a43-9b5f-19a83d64d169-kube-api-access-2j96j\") pod \"payload-pre-processing-e2e-inf-b-e412d5-5ff4cd49ff-8fb8p\" (UID: \"5ff70346-02a9-4a43-9b5f-19a83d64d169\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e412d5-5ff4cd49ff-8fb8p" Nov 02 01:39:22.355449 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:22.348323 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f3cd62e-2f33-481d-a827-63d2fddf6a46-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e412d5-68746dfdfc-lrhnf\" (UID: \"8f3cd62e-2f33-481d-a827-63d2fddf6a46\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e412d5-68746dfdfc-lrhnf" Nov 02 01:39:22.410400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:22.385337 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ff70346-02a9-4a43-9b5f-19a83d64d169-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e412d5-5ff4cd49ff-8fb8p\" (UID: \"5ff70346-02a9-4a43-9b5f-19a83d64d169\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e412d5-5ff4cd49ff-8fb8p" Nov 02 01:39:22.411336 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:22.409239 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s6kj\" (UniqueName: \"kubernetes.io/projected/8f3cd62e-2f33-481d-a827-63d2fddf6a46-kube-api-access-9s6kj\") pod \"payload-processing-e2e-inf-b-e412d5-68746dfdfc-lrhnf\" (UID: \"8f3cd62e-2f33-481d-a827-63d2fddf6a46\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e412d5-68746dfdfc-lrhnf" Nov 02 01:39:22.457800 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:22.457769 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp"] Nov 02 01:39:22.559727 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:22.531260 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j96j\" (UniqueName: \"kubernetes.io/projected/5ff70346-02a9-4a43-9b5f-19a83d64d169-kube-api-access-2j96j\") pod \"payload-pre-processing-e2e-inf-b-e412d5-5ff4cd49ff-8fb8p\" (UID: \"5ff70346-02a9-4a43-9b5f-19a83d64d169\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e412d5-5ff4cd49ff-8fb8p" Nov 02 01:39:22.604479 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:22.593952 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e412d5-68746dfdfc-lrhnf" Nov 02 01:39:22.604479 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:22.595126 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp"] Nov 02 01:39:22.604479 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:22.595286 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp" Nov 02 01:39:22.623490 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:22.617459 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-bf1e55-openshift-default-dockercfg-w8kcb\"" Nov 02 01:39:22.667399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:22.660426 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c53652cf-999e-4b99-b2f7-e779912799ce-workload-certs\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp\" (UID: \"c53652cf-999e-4b99-b2f7-e779912799ce\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp" Nov 02 01:39:22.667399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:22.660535 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c53652cf-999e-4b99-b2f7-e779912799ce-istio-envoy\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp\" (UID: \"c53652cf-999e-4b99-b2f7-e779912799ce\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp" Nov 02 01:39:22.667399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:22.660596 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c53652cf-999e-4b99-b2f7-e779912799ce-istio-data\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp\" (UID: \"c53652cf-999e-4b99-b2f7-e779912799ce\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp" Nov 02 01:39:22.667399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:22.660704 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c53652cf-999e-4b99-b2f7-e779912799ce-istio-token\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp\" (UID: \"c53652cf-999e-4b99-b2f7-e779912799ce\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp" Nov 02 01:39:22.667399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:22.660899 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c53652cf-999e-4b99-b2f7-e779912799ce-credential-socket\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp\" (UID: \"c53652cf-999e-4b99-b2f7-e779912799ce\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp" Nov 02 01:39:22.667399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:22.660996 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c53652cf-999e-4b99-b2f7-e779912799ce-workload-socket\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp\" (UID: \"c53652cf-999e-4b99-b2f7-e779912799ce\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp" Nov 02 01:39:22.667399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:22.661126 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptclz\" (UniqueName: \"kubernetes.io/projected/c53652cf-999e-4b99-b2f7-e779912799ce-kube-api-access-ptclz\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp\" (UID: \"c53652cf-999e-4b99-b2f7-e779912799ce\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp" Nov 02 01:39:22.667399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:22.661391 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e412d5-5ff4cd49ff-8fb8p" Nov 02 01:39:22.685461 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:22.668590 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c53652cf-999e-4b99-b2f7-e779912799ce-istiod-ca-cert\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp\" (UID: \"c53652cf-999e-4b99-b2f7-e779912799ce\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp" Nov 02 01:39:22.685461 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:22.668721 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c53652cf-999e-4b99-b2f7-e779912799ce-istio-podinfo\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp\" (UID: \"c53652cf-999e-4b99-b2f7-e779912799ce\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp" Nov 02 01:39:22.789549 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:22.784774 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c53652cf-999e-4b99-b2f7-e779912799ce-workload-socket\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp\" (UID: \"c53652cf-999e-4b99-b2f7-e779912799ce\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp" Nov 02 01:39:22.789549 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:22.784878 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptclz\" (UniqueName: \"kubernetes.io/projected/c53652cf-999e-4b99-b2f7-e779912799ce-kube-api-access-ptclz\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp\" (UID: \"c53652cf-999e-4b99-b2f7-e779912799ce\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp" Nov 02 01:39:22.789549 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:22.784971 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c53652cf-999e-4b99-b2f7-e779912799ce-istiod-ca-cert\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp\" (UID: \"c53652cf-999e-4b99-b2f7-e779912799ce\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp" Nov 02 01:39:22.789549 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:22.785063 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c53652cf-999e-4b99-b2f7-e779912799ce-istio-podinfo\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp\" (UID: \"c53652cf-999e-4b99-b2f7-e779912799ce\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp" Nov 02 01:39:22.789549 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:22.785190 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c53652cf-999e-4b99-b2f7-e779912799ce-workload-certs\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp\" (UID: \"c53652cf-999e-4b99-b2f7-e779912799ce\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp" Nov 02 01:39:22.789549 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:22.785266 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c53652cf-999e-4b99-b2f7-e779912799ce-istio-envoy\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp\" (UID: \"c53652cf-999e-4b99-b2f7-e779912799ce\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp" Nov 02 01:39:22.789549 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:22.785309 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c53652cf-999e-4b99-b2f7-e779912799ce-istio-data\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp\" (UID: \"c53652cf-999e-4b99-b2f7-e779912799ce\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp" Nov 02 01:39:22.789549 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:22.785365 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c53652cf-999e-4b99-b2f7-e779912799ce-istio-token\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp\" (UID: \"c53652cf-999e-4b99-b2f7-e779912799ce\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp" Nov 02 01:39:22.789549 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:22.785506 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c53652cf-999e-4b99-b2f7-e779912799ce-credential-socket\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp\" (UID: \"c53652cf-999e-4b99-b2f7-e779912799ce\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp" Nov 02 01:39:22.789549 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:22.785997 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c53652cf-999e-4b99-b2f7-e779912799ce-credential-socket\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp\" (UID: \"c53652cf-999e-4b99-b2f7-e779912799ce\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp" Nov 02 01:39:22.789549 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:22.786289 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c53652cf-999e-4b99-b2f7-e779912799ce-workload-socket\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp\" (UID: \"c53652cf-999e-4b99-b2f7-e779912799ce\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp" Nov 02 01:39:22.789549 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:22.786840 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c53652cf-999e-4b99-b2f7-e779912799ce-workload-certs\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp\" (UID: \"c53652cf-999e-4b99-b2f7-e779912799ce\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp" Nov 02 01:39:22.789549 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:22.787683 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c53652cf-999e-4b99-b2f7-e779912799ce-istio-data\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp\" (UID: \"c53652cf-999e-4b99-b2f7-e779912799ce\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp" Nov 02 01:39:22.789549 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:22.789283 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c53652cf-999e-4b99-b2f7-e779912799ce-istiod-ca-cert\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp\" (UID: \"c53652cf-999e-4b99-b2f7-e779912799ce\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp" Nov 02 01:39:22.791335 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:22.791282 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c53652cf-999e-4b99-b2f7-e779912799ce-istio-envoy\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp\" (UID: \"c53652cf-999e-4b99-b2f7-e779912799ce\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp" Nov 02 01:39:22.794826 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:22.794780 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c53652cf-999e-4b99-b2f7-e779912799ce-istio-podinfo\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp\" (UID: \"c53652cf-999e-4b99-b2f7-e779912799ce\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp" Nov 02 01:39:22.819943 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:22.819858 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c53652cf-999e-4b99-b2f7-e779912799ce-istio-token\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp\" (UID: \"c53652cf-999e-4b99-b2f7-e779912799ce\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp" Nov 02 01:39:22.887400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:22.883665 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptclz\" (UniqueName: \"kubernetes.io/projected/c53652cf-999e-4b99-b2f7-e779912799ce-kube-api-access-ptclz\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp\" (UID: \"c53652cf-999e-4b99-b2f7-e779912799ce\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp" Nov 02 01:39:23.083410 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:23.064566 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp" Nov 02 01:39:23.135348 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:23.135148 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e412d5-5ff4cd49ff-8fb8p"] Nov 02 01:39:23.197671 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:23.197277 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e412d5-68746dfdfc-lrhnf"] Nov 02 01:39:23.442056 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:39:23.442020 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc53652cf_999e_4b99_b2f7_e779912799ce.slice/crio-6433ce90e1b55a9ea7157c84b47a4f04d69e519b08a2de6e25def154048752a5 WatchSource:0}: Error finding container 6433ce90e1b55a9ea7157c84b47a4f04d69e519b08a2de6e25def154048752a5: Status 404 returned error can't find the container with id 6433ce90e1b55a9ea7157c84b47a4f04d69e519b08a2de6e25def154048752a5 Nov 02 01:39:23.442492 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:23.442462 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp"] Nov 02 01:39:23.445867 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:23.445828 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:39:23.445977 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:23.445899 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:39:23.445977 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:23.445929 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:39:24.192413 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:24.191761 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp" event={"ID":"c53652cf-999e-4b99-b2f7-e779912799ce","Type":"ContainerStarted","Data":"94b6875f8e746e059d65e1b3ea6995d7a8bf6fc3fa03b93c44d13259dcd7901b"} Nov 02 01:39:24.192413 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:24.191856 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp" event={"ID":"c53652cf-999e-4b99-b2f7-e779912799ce","Type":"ContainerStarted","Data":"6433ce90e1b55a9ea7157c84b47a4f04d69e519b08a2de6e25def154048752a5"} Nov 02 01:39:24.209486 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:24.209358 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e412d5-68746dfdfc-lrhnf" event={"ID":"8f3cd62e-2f33-481d-a827-63d2fddf6a46","Type":"ContainerStarted","Data":"1ef6ed1b4872fbbad3e46a725197598d9776cff39bc991286854717a2b8ccc7a"} Nov 02 01:39:24.209673 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:24.209518 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-e412d5-68746dfdfc-lrhnf" Nov 02 01:39:24.209673 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:24.209547 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e412d5-68746dfdfc-lrhnf" event={"ID":"8f3cd62e-2f33-481d-a827-63d2fddf6a46","Type":"ContainerStarted","Data":"556a34a1708cbb534de86a5e6f2a71ca6a6656575e73e9ed66bef56b970871e7"} Nov 02 01:39:24.214649 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:24.214617 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e412d5-5ff4cd49ff-8fb8p" event={"ID":"5ff70346-02a9-4a43-9b5f-19a83d64d169","Type":"ContainerStarted","Data":"a35b3934809728a190648c59983a82ef5be7457184fa58a425b5c1a013284bcb"} Nov 02 01:39:24.214799 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:24.214674 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e412d5-5ff4cd49ff-8fb8p" event={"ID":"5ff70346-02a9-4a43-9b5f-19a83d64d169","Type":"ContainerStarted","Data":"f439ce7b1b32f47081a1395a6d21355b26b8ad4c5992bd90d2ec9b05ff6e006e"} Nov 02 01:39:24.214859 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:24.214806 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e412d5-5ff4cd49ff-8fb8p" Nov 02 01:39:24.225449 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:24.225386 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp" podStartSLOduration=2.225354285 podStartE2EDuration="2.225354285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:39:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:39:24.222524594 +0000 UTC m=+1468.843529806" watchObservedRunningTime="2026-11-02 01:39:24.225354285 +0000 UTC m=+1468.846359496" Nov 02 01:39:24.249309 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:24.248653 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-e412d5-68746dfdfc-lrhnf" podStartSLOduration=2.2486343030000002 podStartE2EDuration="2.248634303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:39:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:39:24.243870399 +0000 UTC m=+1468.864875612" watchObservedRunningTime="2026-11-02 01:39:24.248634303 +0000 UTC m=+1468.869639517" Nov 02 01:39:24.298205 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:24.298138 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e412d5-5ff4cd49ff-8fb8p" podStartSLOduration=3.298114387 podStartE2EDuration="3.298114387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:39:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:39:24.282960616 +0000 UTC m=+1468.903965837" watchObservedRunningTime="2026-11-02 01:39:24.298114387 +0000 UTC m=+1468.919119599" Nov 02 01:39:25.077398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:25.065672 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp" Nov 02 01:39:25.089706 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:25.089660 4427 patch_prober.go:29] interesting pod/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Nov 02 01:39:25.089931 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:25.089730 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp" podUID="c53652cf-999e-4b99-b2f7-e779912799ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Nov 02 01:39:26.070927 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:26.069881 4427 patch_prober.go:29] interesting pod/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Nov 02 01:39:26.070927 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:26.069946 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp" podUID="c53652cf-999e-4b99-b2f7-e779912799ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Nov 02 01:39:27.080401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:27.065770 4427 patch_prober.go:29] interesting pod/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Nov 02 01:39:27.080401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:27.065851 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp" podUID="c53652cf-999e-4b99-b2f7-e779912799ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Nov 02 01:39:27.295234 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:27.295200 4427 generic.go:356] "Generic (PLEG): container finished" podID="331a730d-c524-4f68-9360-25e29df1ffb4" containerID="57486b9e30ae367c16b1c917c510c2aad33b96025e35d27ba63d680d0d2e82ed" exitCode=0 Nov 02 01:39:27.295445 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:27.295266 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n" event={"ID":"331a730d-c524-4f68-9360-25e29df1ffb4","Type":"ContainerDied","Data":"57486b9e30ae367c16b1c917c510c2aad33b96025e35d27ba63d680d0d2e82ed"} Nov 02 01:39:27.296270 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:27.296248 4427 scope.go:122] "RemoveContainer" containerID="57486b9e30ae367c16b1c917c510c2aad33b96025e35d27ba63d680d0d2e82ed" Nov 02 01:39:27.297991 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:27.297806 4427 generic.go:356] "Generic (PLEG): container finished" podID="95975531-e9c3-4fcb-9f93-ddab47bb62ba" containerID="e75bccb167e5e1f6882788fe66536261d715bbce3d95a0d95191a10c531b4863" exitCode=0 Nov 02 01:39:27.297991 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:27.297871 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h" event={"ID":"95975531-e9c3-4fcb-9f93-ddab47bb62ba","Type":"ContainerDied","Data":"e75bccb167e5e1f6882788fe66536261d715bbce3d95a0d95191a10c531b4863"} Nov 02 01:39:27.299824 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:27.298495 4427 scope.go:122] "RemoveContainer" containerID="e75bccb167e5e1f6882788fe66536261d715bbce3d95a0d95191a10c531b4863" Nov 02 01:39:27.301056 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:27.301024 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:39:27.301179 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:27.301112 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:39:27.301179 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:27.301158 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:39:27.302109 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:27.301743 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:39:27.302109 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:27.301839 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:39:27.302109 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:27.301886 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:39:28.081396 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:28.080826 4427 patch_prober.go:29] interesting pod/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Nov 02 01:39:28.081396 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:28.080894 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp" podUID="c53652cf-999e-4b99-b2f7-e779912799ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Nov 02 01:39:28.310155 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:28.310116 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n" event={"ID":"331a730d-c524-4f68-9360-25e29df1ffb4","Type":"ContainerStarted","Data":"337ba89cb083e16b593a211c66c84d79bf4ba3e895b894306f4a94e4f29e00db"} Nov 02 01:39:28.312548 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:28.312519 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h" event={"ID":"95975531-e9c3-4fcb-9f93-ddab47bb62ba","Type":"ContainerStarted","Data":"67f8e1110ac77d3bcdbc8748e701a2e277e503141cc9103464ad79bec919c73c"} Nov 02 01:39:29.006610 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:29.005220 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h" Nov 02 01:39:29.006946 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:29.006919 4427 patch_prober.go:29] interesting pod/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 02 01:39:29.007130 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:29.007104 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h" podUID="95975531-e9c3-4fcb-9f93-ddab47bb62ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 02 01:39:29.081403 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:29.067812 4427 patch_prober.go:29] interesting pod/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Nov 02 01:39:29.081403 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:29.067877 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp" podUID="c53652cf-999e-4b99-b2f7-e779912799ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Nov 02 01:39:29.091076 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:29.088904 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n" Nov 02 01:39:29.111398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:29.095535 4427 patch_prober.go:29] interesting pod/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 02 01:39:29.111398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:29.095602 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n" podUID="331a730d-c524-4f68-9360-25e29df1ffb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 02 01:39:30.011401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:30.010568 4427 patch_prober.go:29] interesting pod/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 02 01:39:30.011401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:30.010624 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h" podUID="95975531-e9c3-4fcb-9f93-ddab47bb62ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 02 01:39:30.067400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:30.066820 4427 patch_prober.go:29] interesting pod/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Nov 02 01:39:30.067400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:30.066882 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp" podUID="c53652cf-999e-4b99-b2f7-e779912799ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Nov 02 01:39:30.093688 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:30.093654 4427 patch_prober.go:29] interesting pod/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 02 01:39:30.094310 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:30.094259 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n" podUID="331a730d-c524-4f68-9360-25e29df1ffb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 02 01:39:30.634414 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:30.634383 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h" Nov 02 01:39:31.005457 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:31.003533 4427 patch_prober.go:29] interesting pod/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 02 01:39:31.005457 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:31.003597 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h" podUID="95975531-e9c3-4fcb-9f93-ddab47bb62ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 02 01:39:31.021696 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:31.021663 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n" Nov 02 01:39:31.084245 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:31.084210 4427 patch_prober.go:29] interesting pod/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 02 01:39:31.084549 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:31.084523 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n" podUID="331a730d-c524-4f68-9360-25e29df1ffb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 02 01:39:31.097730 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:31.097701 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp" Nov 02 01:39:31.100034 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:31.100013 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp" Nov 02 01:39:31.108416 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:31.108352 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp" Nov 02 01:39:31.637356 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:39:31.637308 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:39:32.000909 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:32.000812 4427 patch_prober.go:29] interesting pod/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 02 01:39:32.001226 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:32.001191 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h" podUID="95975531-e9c3-4fcb-9f93-ddab47bb62ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 02 01:39:32.086325 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:32.085840 4427 patch_prober.go:29] interesting pod/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 02 01:39:32.086325 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:32.085905 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n" podUID="331a730d-c524-4f68-9360-25e29df1ffb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 02 01:39:33.030901 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:33.030853 4427 patch_prober.go:29] interesting pod/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 02 01:39:33.031502 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:33.030932 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h" podUID="95975531-e9c3-4fcb-9f93-ddab47bb62ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 02 01:39:33.090862 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:33.090822 4427 patch_prober.go:29] interesting pod/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 02 01:39:33.091179 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:33.091149 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n" podUID="331a730d-c524-4f68-9360-25e29df1ffb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 02 01:39:34.008404 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:34.006766 4427 patch_prober.go:29] interesting pod/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 02 01:39:34.008404 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:34.006827 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h" podUID="95975531-e9c3-4fcb-9f93-ddab47bb62ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 02 01:39:34.088995 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:34.088946 4427 patch_prober.go:29] interesting pod/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 02 01:39:34.089664 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:34.089612 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n" podUID="331a730d-c524-4f68-9360-25e29df1ffb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 02 01:39:35.005684 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:35.005649 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h" Nov 02 01:39:35.006839 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:35.006816 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h" Nov 02 01:39:35.092620 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:35.092588 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n" Nov 02 01:39:35.095071 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:35.094928 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n" Nov 02 01:39:35.254402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:35.253632 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e412d5-5ff4cd49ff-8fb8p" Nov 02 01:39:35.257631 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:39:35.257536 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-e412d5-68746dfdfc-lrhnf" Nov 02 01:40:01.587746 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:40:01.587701 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:40:31.585921 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:40:31.585887 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:40:48.904502 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:40:48.904463 4427 patch_prober.go:29] interesting pod/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": read tcp 10.133.0.2:58800->10.133.0.132:15021: read: connection reset by peer" start-of-body= Nov 02 01:40:48.905004 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:40:48.904559 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w" podUID="51776e18-2d31-454f-8c33-d7fbb169c601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": read tcp 10.133.0.2:58800->10.133.0.132:15021: read: connection reset by peer" Nov 02 01:40:49.510387 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:40:49.510334 4427 generic.go:356] "Generic (PLEG): container finished" podID="51776e18-2d31-454f-8c33-d7fbb169c601" containerID="40531b877859708237bf174c5ce262ff352f62c3d4a21cbd22017be0ad9db7ca" exitCode=0 Nov 02 01:40:49.510564 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:40:49.510426 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w" event={"ID":"51776e18-2d31-454f-8c33-d7fbb169c601","Type":"ContainerDied","Data":"40531b877859708237bf174c5ce262ff352f62c3d4a21cbd22017be0ad9db7ca"} Nov 02 01:40:49.510960 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:40:49.510945 4427 scope.go:122] "RemoveContainer" containerID="40531b877859708237bf174c5ce262ff352f62c3d4a21cbd22017be0ad9db7ca" Nov 02 01:40:49.513311 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:40:49.513283 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:40:49.513435 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:40:49.513361 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:40:49.513435 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:40:49.513423 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:40:50.531868 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:40:50.531827 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w" event={"ID":"51776e18-2d31-454f-8c33-d7fbb169c601","Type":"ContainerStarted","Data":"d54bd0fa45fe5c2c3e233a10fa99a4c2028bc5b352bf4cb393d2d486a7783358"} Nov 02 01:40:51.245046 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:40:51.245013 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w" Nov 02 01:40:51.245215 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:40:51.245180 4427 patch_prober.go:29] interesting pod/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 02 01:40:51.245261 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:40:51.245243 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w" podUID="51776e18-2d31-454f-8c33-d7fbb169c601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 02 01:40:52.244341 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:40:52.244298 4427 patch_prober.go:29] interesting pod/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 02 01:40:52.244804 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:40:52.244361 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w" podUID="51776e18-2d31-454f-8c33-d7fbb169c601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 02 01:40:53.244294 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:40:53.244244 4427 patch_prober.go:29] interesting pod/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 02 01:40:53.244509 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:40:53.244305 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w" podUID="51776e18-2d31-454f-8c33-d7fbb169c601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 02 01:40:54.244483 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:40:54.244433 4427 patch_prober.go:29] interesting pod/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 02 01:40:54.244721 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:40:54.244514 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w" podUID="51776e18-2d31-454f-8c33-d7fbb169c601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 02 01:40:55.246279 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:40:55.244788 4427 patch_prober.go:29] interesting pod/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 02 01:40:55.246279 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:40:55.244851 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w" podUID="51776e18-2d31-454f-8c33-d7fbb169c601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 02 01:40:55.313626 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:40:55.313575 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl"] Nov 02 01:40:55.314363 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:40:55.314294 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl" podUID="282522e6-7d16-4316-bf07-5e2706155435" containerName="istio-proxy" containerID="cri-o://3fc39fb8b747ca34e53fcc1003d509ec7f0fcdaa6cbf66eeccf80d36e47caaa5" gracePeriod=30 Nov 02 01:40:56.250117 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:40:56.250064 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w" Nov 02 01:40:56.250833 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:40:56.250604 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w" Nov 02 01:40:56.251462 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:40:56.251446 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w" Nov 02 01:40:57.056266 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:40:57.056232 4427 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 01:40:57.056475 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:40:57.056294 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl" podUID="282522e6-7d16-4316-bf07-5e2706155435" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 01:41:00.639835 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.639806 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl" Nov 02 01:41:00.671662 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.671622 4427 generic.go:356] "Generic (PLEG): container finished" podID="282522e6-7d16-4316-bf07-5e2706155435" containerID="3fc39fb8b747ca34e53fcc1003d509ec7f0fcdaa6cbf66eeccf80d36e47caaa5" exitCode=0 Nov 02 01:41:00.671896 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.671737 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl" Nov 02 01:41:00.672222 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.672192 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl" event={"ID":"282522e6-7d16-4316-bf07-5e2706155435","Type":"ContainerDied","Data":"3fc39fb8b747ca34e53fcc1003d509ec7f0fcdaa6cbf66eeccf80d36e47caaa5"} Nov 02 01:41:00.672325 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.672242 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl" event={"ID":"282522e6-7d16-4316-bf07-5e2706155435","Type":"ContainerDied","Data":"b14d07df9a7e5f68cb94b3fd1fcbece2e9ad3fb58c196818fa2bb5f3bc8d04b4"} Nov 02 01:41:00.672325 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.672272 4427 scope.go:122] "RemoveContainer" containerID="3fc39fb8b747ca34e53fcc1003d509ec7f0fcdaa6cbf66eeccf80d36e47caaa5" Nov 02 01:41:00.678219 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.678190 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt"] Nov 02 01:41:00.679532 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.679193 4427 cpu_manager.go:414] "containerMap: removing container" podUID="282522e6-7d16-4316-bf07-5e2706155435" containerName="istio-proxy" Nov 02 01:41:00.679532 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.679216 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="282522e6-7d16-4316-bf07-5e2706155435" containerName="istio-proxy" Nov 02 01:41:00.679532 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.679264 4427 cpu_manager.go:414] "containerMap: removing container" podUID="282522e6-7d16-4316-bf07-5e2706155435" containerName="istio-proxy" Nov 02 01:41:00.679532 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.679270 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="282522e6-7d16-4316-bf07-5e2706155435" containerName="istio-proxy" Nov 02 01:41:00.679532 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.679492 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="282522e6-7d16-4316-bf07-5e2706155435" containerName="istio-proxy" Nov 02 01:41:00.679532 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.679515 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="282522e6-7d16-4316-bf07-5e2706155435" containerName="istio-proxy" Nov 02 01:41:00.683875 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.683851 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt" Nov 02 01:41:00.688533 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.688483 4427 scope.go:122] "RemoveContainer" containerID="3e4e3e4518a25da89666056534b7618c728efc423e571f4d498aa3205516d6ef" Nov 02 01:41:00.692732 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.691481 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-f689bc-openshift-default-dockercfg-zlxb6\"" Nov 02 01:41:00.702545 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.702138 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt"] Nov 02 01:41:00.720391 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.720352 4427 scope.go:122] "RemoveContainer" containerID="3fc39fb8b747ca34e53fcc1003d509ec7f0fcdaa6cbf66eeccf80d36e47caaa5" Nov 02 01:41:00.720827 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:41:00.720795 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fc39fb8b747ca34e53fcc1003d509ec7f0fcdaa6cbf66eeccf80d36e47caaa5\": container with ID starting with 3fc39fb8b747ca34e53fcc1003d509ec7f0fcdaa6cbf66eeccf80d36e47caaa5 not found: ID does not exist" containerID="3fc39fb8b747ca34e53fcc1003d509ec7f0fcdaa6cbf66eeccf80d36e47caaa5" Nov 02 01:41:00.720952 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.720834 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fc39fb8b747ca34e53fcc1003d509ec7f0fcdaa6cbf66eeccf80d36e47caaa5"} err="failed to get container status \"3fc39fb8b747ca34e53fcc1003d509ec7f0fcdaa6cbf66eeccf80d36e47caaa5\": rpc error: code = NotFound desc = could not find container \"3fc39fb8b747ca34e53fcc1003d509ec7f0fcdaa6cbf66eeccf80d36e47caaa5\": container with ID starting with 3fc39fb8b747ca34e53fcc1003d509ec7f0fcdaa6cbf66eeccf80d36e47caaa5 not found: ID does not exist" Nov 02 01:41:00.720952 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.720851 4427 scope.go:122] "RemoveContainer" containerID="3e4e3e4518a25da89666056534b7618c728efc423e571f4d498aa3205516d6ef" Nov 02 01:41:00.721184 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:41:00.721142 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e4e3e4518a25da89666056534b7618c728efc423e571f4d498aa3205516d6ef\": container with ID starting with 3e4e3e4518a25da89666056534b7618c728efc423e571f4d498aa3205516d6ef not found: ID does not exist" containerID="3e4e3e4518a25da89666056534b7618c728efc423e571f4d498aa3205516d6ef" Nov 02 01:41:00.721256 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.721193 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e4e3e4518a25da89666056534b7618c728efc423e571f4d498aa3205516d6ef"} err="failed to get container status \"3e4e3e4518a25da89666056534b7618c728efc423e571f4d498aa3205516d6ef\": rpc error: code = NotFound desc = could not find container \"3e4e3e4518a25da89666056534b7618c728efc423e571f4d498aa3205516d6ef\": container with ID starting with 3e4e3e4518a25da89666056534b7618c728efc423e571f4d498aa3205516d6ef not found: ID does not exist" Nov 02 01:41:00.746262 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.746226 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/282522e6-7d16-4316-bf07-5e2706155435-istio-token\" (UniqueName: \"kubernetes.io/projected/282522e6-7d16-4316-bf07-5e2706155435-istio-token\") pod \"282522e6-7d16-4316-bf07-5e2706155435\" (UID: \"282522e6-7d16-4316-bf07-5e2706155435\") " Nov 02 01:41:00.746479 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.746284 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/282522e6-7d16-4316-bf07-5e2706155435-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/282522e6-7d16-4316-bf07-5e2706155435-workload-certs\") pod \"282522e6-7d16-4316-bf07-5e2706155435\" (UID: \"282522e6-7d16-4316-bf07-5e2706155435\") " Nov 02 01:41:00.746479 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.746339 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/282522e6-7d16-4316-bf07-5e2706155435-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/282522e6-7d16-4316-bf07-5e2706155435-istio-podinfo\") pod \"282522e6-7d16-4316-bf07-5e2706155435\" (UID: \"282522e6-7d16-4316-bf07-5e2706155435\") " Nov 02 01:41:00.746479 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.746461 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/282522e6-7d16-4316-bf07-5e2706155435-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/282522e6-7d16-4316-bf07-5e2706155435-istiod-ca-cert\") pod \"282522e6-7d16-4316-bf07-5e2706155435\" (UID: \"282522e6-7d16-4316-bf07-5e2706155435\") " Nov 02 01:41:00.746667 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.746508 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/282522e6-7d16-4316-bf07-5e2706155435-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/282522e6-7d16-4316-bf07-5e2706155435-credential-socket\") pod \"282522e6-7d16-4316-bf07-5e2706155435\" (UID: \"282522e6-7d16-4316-bf07-5e2706155435\") " Nov 02 01:41:00.746667 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.746557 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/282522e6-7d16-4316-bf07-5e2706155435-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/282522e6-7d16-4316-bf07-5e2706155435-workload-socket\") pod \"282522e6-7d16-4316-bf07-5e2706155435\" (UID: \"282522e6-7d16-4316-bf07-5e2706155435\") " Nov 02 01:41:00.746667 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.746596 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/282522e6-7d16-4316-bf07-5e2706155435-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/282522e6-7d16-4316-bf07-5e2706155435-istio-envoy\") pod \"282522e6-7d16-4316-bf07-5e2706155435\" (UID: \"282522e6-7d16-4316-bf07-5e2706155435\") " Nov 02 01:41:00.746866 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.746757 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/282522e6-7d16-4316-bf07-5e2706155435-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/282522e6-7d16-4316-bf07-5e2706155435-istio-data\") pod \"282522e6-7d16-4316-bf07-5e2706155435\" (UID: \"282522e6-7d16-4316-bf07-5e2706155435\") " Nov 02 01:41:00.746959 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.746939 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/282522e6-7d16-4316-bf07-5e2706155435-kube-api-access-rdq8p\" (UniqueName: \"kubernetes.io/projected/282522e6-7d16-4316-bf07-5e2706155435-kube-api-access-rdq8p\") pod \"282522e6-7d16-4316-bf07-5e2706155435\" (UID: \"282522e6-7d16-4316-bf07-5e2706155435\") " Nov 02 01:41:00.746959 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.746945 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/282522e6-7d16-4316-bf07-5e2706155435-credential-socket" pod "282522e6-7d16-4316-bf07-5e2706155435" (UID: "282522e6-7d16-4316-bf07-5e2706155435"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:41:00.747299 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.747265 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/282522e6-7d16-4316-bf07-5e2706155435-workload-socket" pod "282522e6-7d16-4316-bf07-5e2706155435" (UID: "282522e6-7d16-4316-bf07-5e2706155435"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:41:00.748563 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.747891 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/282522e6-7d16-4316-bf07-5e2706155435-istiod-ca-cert" pod "282522e6-7d16-4316-bf07-5e2706155435" (UID: "282522e6-7d16-4316-bf07-5e2706155435"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:41:00.748563 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.748141 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/282522e6-7d16-4316-bf07-5e2706155435-istio-data" pod "282522e6-7d16-4316-bf07-5e2706155435" (UID: "282522e6-7d16-4316-bf07-5e2706155435"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:41:00.749244 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.749218 4427 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/282522e6-7d16-4316-bf07-5e2706155435-credential-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:00.749363 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.749250 4427 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/282522e6-7d16-4316-bf07-5e2706155435-workload-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:00.749363 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.749265 4427 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/282522e6-7d16-4316-bf07-5e2706155435-istio-data\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:00.749363 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.749279 4427 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/282522e6-7d16-4316-bf07-5e2706155435-istiod-ca-cert\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:00.749700 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.749678 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/282522e6-7d16-4316-bf07-5e2706155435-workload-certs" pod "282522e6-7d16-4316-bf07-5e2706155435" (UID: "282522e6-7d16-4316-bf07-5e2706155435"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:41:00.756734 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.756708 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/282522e6-7d16-4316-bf07-5e2706155435-istio-envoy" pod "282522e6-7d16-4316-bf07-5e2706155435" (UID: "282522e6-7d16-4316-bf07-5e2706155435"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:41:00.756937 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.756911 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/282522e6-7d16-4316-bf07-5e2706155435-istio-token" pod "282522e6-7d16-4316-bf07-5e2706155435" (UID: "282522e6-7d16-4316-bf07-5e2706155435"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:41:00.756996 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.756958 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/282522e6-7d16-4316-bf07-5e2706155435-kube-api-access-rdq8p" pod "282522e6-7d16-4316-bf07-5e2706155435" (UID: "282522e6-7d16-4316-bf07-5e2706155435"). InnerVolumeSpecName "kube-api-access-rdq8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:41:00.758160 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.758139 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/282522e6-7d16-4316-bf07-5e2706155435-istio-podinfo" pod "282522e6-7d16-4316-bf07-5e2706155435" (UID: "282522e6-7d16-4316-bf07-5e2706155435"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:41:00.858936 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.852673 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82d82a81-f1df-446d-b07b-20f510354a7e-istiod-ca-cert\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt\" (UID: \"82d82a81-f1df-446d-b07b-20f510354a7e\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt" Nov 02 01:41:00.858936 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.852831 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82d82a81-f1df-446d-b07b-20f510354a7e-workload-certs\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt\" (UID: \"82d82a81-f1df-446d-b07b-20f510354a7e\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt" Nov 02 01:41:00.858936 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.852871 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82d82a81-f1df-446d-b07b-20f510354a7e-istio-data\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt\" (UID: \"82d82a81-f1df-446d-b07b-20f510354a7e\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt" Nov 02 01:41:00.858936 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.852941 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82d82a81-f1df-446d-b07b-20f510354a7e-istio-token\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt\" (UID: \"82d82a81-f1df-446d-b07b-20f510354a7e\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt" Nov 02 01:41:00.890411 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.875162 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82d82a81-f1df-446d-b07b-20f510354a7e-istio-envoy\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt\" (UID: \"82d82a81-f1df-446d-b07b-20f510354a7e\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt" Nov 02 01:41:00.890411 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.875442 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82d82a81-f1df-446d-b07b-20f510354a7e-workload-socket\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt\" (UID: \"82d82a81-f1df-446d-b07b-20f510354a7e\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt" Nov 02 01:41:00.890411 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.875554 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82d82a81-f1df-446d-b07b-20f510354a7e-istio-podinfo\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt\" (UID: \"82d82a81-f1df-446d-b07b-20f510354a7e\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt" Nov 02 01:41:00.890411 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.875602 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82d82a81-f1df-446d-b07b-20f510354a7e-credential-socket\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt\" (UID: \"82d82a81-f1df-446d-b07b-20f510354a7e\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt" Nov 02 01:41:00.890411 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.875648 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mzz6\" (UniqueName: \"kubernetes.io/projected/82d82a81-f1df-446d-b07b-20f510354a7e-kube-api-access-7mzz6\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt\" (UID: \"82d82a81-f1df-446d-b07b-20f510354a7e\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt" Nov 02 01:41:00.890411 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.875808 4427 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/282522e6-7d16-4316-bf07-5e2706155435-istio-token\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:00.890411 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.875824 4427 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/282522e6-7d16-4316-bf07-5e2706155435-workload-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:00.890411 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.875838 4427 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/282522e6-7d16-4316-bf07-5e2706155435-istio-podinfo\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:00.890411 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.876121 4427 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/282522e6-7d16-4316-bf07-5e2706155435-istio-envoy\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:00.890411 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.876147 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdq8p\" (UniqueName: \"kubernetes.io/projected/282522e6-7d16-4316-bf07-5e2706155435-kube-api-access-rdq8p\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:00.977353 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.977316 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82d82a81-f1df-446d-b07b-20f510354a7e-workload-socket\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt\" (UID: \"82d82a81-f1df-446d-b07b-20f510354a7e\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt" Nov 02 01:41:00.977557 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.977418 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82d82a81-f1df-446d-b07b-20f510354a7e-istio-podinfo\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt\" (UID: \"82d82a81-f1df-446d-b07b-20f510354a7e\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt" Nov 02 01:41:00.977557 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.977451 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82d82a81-f1df-446d-b07b-20f510354a7e-credential-socket\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt\" (UID: \"82d82a81-f1df-446d-b07b-20f510354a7e\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt" Nov 02 01:41:00.977557 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.977481 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mzz6\" (UniqueName: \"kubernetes.io/projected/82d82a81-f1df-446d-b07b-20f510354a7e-kube-api-access-7mzz6\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt\" (UID: \"82d82a81-f1df-446d-b07b-20f510354a7e\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt" Nov 02 01:41:00.977716 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.977569 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82d82a81-f1df-446d-b07b-20f510354a7e-istiod-ca-cert\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt\" (UID: \"82d82a81-f1df-446d-b07b-20f510354a7e\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt" Nov 02 01:41:00.977716 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.977688 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82d82a81-f1df-446d-b07b-20f510354a7e-workload-certs\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt\" (UID: \"82d82a81-f1df-446d-b07b-20f510354a7e\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt" Nov 02 01:41:00.977821 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.977723 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82d82a81-f1df-446d-b07b-20f510354a7e-istio-data\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt\" (UID: \"82d82a81-f1df-446d-b07b-20f510354a7e\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt" Nov 02 01:41:00.977821 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.977737 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82d82a81-f1df-446d-b07b-20f510354a7e-workload-socket\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt\" (UID: \"82d82a81-f1df-446d-b07b-20f510354a7e\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt" Nov 02 01:41:00.977821 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.977797 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82d82a81-f1df-446d-b07b-20f510354a7e-credential-socket\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt\" (UID: \"82d82a81-f1df-446d-b07b-20f510354a7e\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt" Nov 02 01:41:00.977821 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.977817 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82d82a81-f1df-446d-b07b-20f510354a7e-istio-token\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt\" (UID: \"82d82a81-f1df-446d-b07b-20f510354a7e\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt" Nov 02 01:41:00.978024 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.977885 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82d82a81-f1df-446d-b07b-20f510354a7e-istio-envoy\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt\" (UID: \"82d82a81-f1df-446d-b07b-20f510354a7e\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt" Nov 02 01:41:00.978146 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.978075 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82d82a81-f1df-446d-b07b-20f510354a7e-istio-data\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt\" (UID: \"82d82a81-f1df-446d-b07b-20f510354a7e\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt" Nov 02 01:41:00.978222 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.978152 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82d82a81-f1df-446d-b07b-20f510354a7e-workload-certs\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt\" (UID: \"82d82a81-f1df-446d-b07b-20f510354a7e\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt" Nov 02 01:41:00.978398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.978352 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82d82a81-f1df-446d-b07b-20f510354a7e-istiod-ca-cert\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt\" (UID: \"82d82a81-f1df-446d-b07b-20f510354a7e\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt" Nov 02 01:41:00.980360 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.980340 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82d82a81-f1df-446d-b07b-20f510354a7e-istio-envoy\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt\" (UID: \"82d82a81-f1df-446d-b07b-20f510354a7e\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt" Nov 02 01:41:00.980574 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.980554 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82d82a81-f1df-446d-b07b-20f510354a7e-istio-podinfo\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt\" (UID: \"82d82a81-f1df-446d-b07b-20f510354a7e\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt" Nov 02 01:41:00.990061 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.990030 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82d82a81-f1df-446d-b07b-20f510354a7e-istio-token\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt\" (UID: \"82d82a81-f1df-446d-b07b-20f510354a7e\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt" Nov 02 01:41:00.990209 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.990190 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mzz6\" (UniqueName: \"kubernetes.io/projected/82d82a81-f1df-446d-b07b-20f510354a7e-kube-api-access-7mzz6\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt\" (UID: \"82d82a81-f1df-446d-b07b-20f510354a7e\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt" Nov 02 01:41:00.998672 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:00.998648 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl"] Nov 02 01:41:01.003473 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:01.003452 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-7lzfl"] Nov 02 01:41:01.021602 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:01.021569 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt" Nov 02 01:41:01.321962 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:01.321928 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt"] Nov 02 01:41:01.322249 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:41:01.322228 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82d82a81_f1df_446d_b07b_20f510354a7e.slice/crio-2e53271e26093c545760bb22ce0188096d60ab876b6883cf401dbc3a0fae75e6 WatchSource:0}: Error finding container 2e53271e26093c545760bb22ce0188096d60ab876b6883cf401dbc3a0fae75e6: Status 404 returned error can't find the container with id 2e53271e26093c545760bb22ce0188096d60ab876b6883cf401dbc3a0fae75e6 Nov 02 01:41:01.326077 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:01.326037 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:41:01.326166 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:01.326126 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:41:01.326215 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:01.326170 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:41:01.538576 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:01.538543 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="282522e6-7d16-4316-bf07-5e2706155435" path="/var/lib/kubelet/pods/282522e6-7d16-4316-bf07-5e2706155435/volumes" Nov 02 01:41:01.590349 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:41:01.590224 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:41:01.736533 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:01.734901 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt" event={"ID":"82d82a81-f1df-446d-b07b-20f510354a7e","Type":"ContainerStarted","Data":"1f7f09782a19394d9cf33bb0bd9e856f43155b286d13912c18e7d0b10098df76"} Nov 02 01:41:01.736533 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:01.735006 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt" event={"ID":"82d82a81-f1df-446d-b07b-20f510354a7e","Type":"ContainerStarted","Data":"2e53271e26093c545760bb22ce0188096d60ab876b6883cf401dbc3a0fae75e6"} Nov 02 01:41:01.763984 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:01.763928 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt" podStartSLOduration=84.763915124 podStartE2EDuration="1m24.763915124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:39:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:41:01.760765348 +0000 UTC m=+1566.381770563" watchObservedRunningTime="2026-11-02 01:41:01.763915124 +0000 UTC m=+1566.384920334" Nov 02 01:41:02.022006 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:02.021973 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt" Nov 02 01:41:02.023470 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:02.023442 4427 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Nov 02 01:41:02.023571 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:02.023495 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt" podUID="82d82a81-f1df-446d-b07b-20f510354a7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Nov 02 01:41:03.022242 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:03.022198 4427 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Nov 02 01:41:03.022782 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:03.022283 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt" podUID="82d82a81-f1df-446d-b07b-20f510354a7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Nov 02 01:41:04.022099 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:04.022061 4427 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Nov 02 01:41:04.022298 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:04.022130 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt" podUID="82d82a81-f1df-446d-b07b-20f510354a7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Nov 02 01:41:05.027398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:05.026975 4427 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Nov 02 01:41:05.027398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:05.027035 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt" podUID="82d82a81-f1df-446d-b07b-20f510354a7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Nov 02 01:41:06.022844 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:06.022806 4427 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Nov 02 01:41:06.023018 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:06.022868 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt" podUID="82d82a81-f1df-446d-b07b-20f510354a7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Nov 02 01:41:07.022150 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:07.022111 4427 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Nov 02 01:41:07.022597 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:07.022173 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt" podUID="82d82a81-f1df-446d-b07b-20f510354a7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Nov 02 01:41:08.022113 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:08.022074 4427 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Nov 02 01:41:08.022304 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:08.022133 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt" podUID="82d82a81-f1df-446d-b07b-20f510354a7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Nov 02 01:41:09.028669 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:09.028635 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt" Nov 02 01:41:09.029131 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:09.029110 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt" Nov 02 01:41:09.030005 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:09.029986 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt" Nov 02 01:41:25.596899 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:25.596865 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2"] Nov 02 01:41:25.597419 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:25.597085 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2" podUID="e1a145f0-821a-4d90-a373-fd65c74c8e57" containerName="istio-proxy" containerID="cri-o://97c5be5723ced963fd89d722494b3b29df47f8c81df7a475b1a5a3dac7b8656a" gracePeriod=30 Nov 02 01:41:25.708557 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:25.708507 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9"] Nov 02 01:41:25.709049 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:25.709017 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9" podUID="19218597-2356-4f9d-bf2b-ac4597da575a" containerName="istio-proxy" containerID="cri-o://93aae06e287d80ec36542db437424297b58b59e34de95c9a40a02a948dfc96d7" gracePeriod=30 Nov 02 01:41:25.795114 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:25.794705 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28"] Nov 02 01:41:25.801121 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:25.801077 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28" podUID="1563efef-2ed3-4347-8399-4c9ca520db4b" containerName="istio-proxy" containerID="cri-o://1c33768a9daa419a56994957b1e1cf5c493d0dcd8cad607d9833533acbe3c5ea" gracePeriod=30 Nov 02 01:41:26.545078 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:26.545035 4427 patch_prober.go:29] interesting pod/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 01:41:26.545262 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:26.545102 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9" podUID="19218597-2356-4f9d-bf2b-ac4597da575a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 01:41:31.088623 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.088593 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2" Nov 02 01:41:31.125240 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.122555 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1a145f0-821a-4d90-a373-fd65c74c8e57-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1a145f0-821a-4d90-a373-fd65c74c8e57-workload-socket\") pod \"e1a145f0-821a-4d90-a373-fd65c74c8e57\" (UID: \"e1a145f0-821a-4d90-a373-fd65c74c8e57\") " Nov 02 01:41:31.125240 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.122602 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1a145f0-821a-4d90-a373-fd65c74c8e57-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1a145f0-821a-4d90-a373-fd65c74c8e57-istiod-ca-cert\") pod \"e1a145f0-821a-4d90-a373-fd65c74c8e57\" (UID: \"e1a145f0-821a-4d90-a373-fd65c74c8e57\") " Nov 02 01:41:31.125240 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.122751 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1a145f0-821a-4d90-a373-fd65c74c8e57-istio-token\" (UniqueName: \"kubernetes.io/projected/e1a145f0-821a-4d90-a373-fd65c74c8e57-istio-token\") pod \"e1a145f0-821a-4d90-a373-fd65c74c8e57\" (UID: \"e1a145f0-821a-4d90-a373-fd65c74c8e57\") " Nov 02 01:41:31.125240 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.122797 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1a145f0-821a-4d90-a373-fd65c74c8e57-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1a145f0-821a-4d90-a373-fd65c74c8e57-workload-certs\") pod \"e1a145f0-821a-4d90-a373-fd65c74c8e57\" (UID: \"e1a145f0-821a-4d90-a373-fd65c74c8e57\") " Nov 02 01:41:31.125240 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.122936 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1a145f0-821a-4d90-a373-fd65c74c8e57-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1a145f0-821a-4d90-a373-fd65c74c8e57-istio-envoy\") pod \"e1a145f0-821a-4d90-a373-fd65c74c8e57\" (UID: \"e1a145f0-821a-4d90-a373-fd65c74c8e57\") " Nov 02 01:41:31.125240 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.122993 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1a145f0-821a-4d90-a373-fd65c74c8e57-kube-api-access-ck6xq\" (UniqueName: \"kubernetes.io/projected/e1a145f0-821a-4d90-a373-fd65c74c8e57-kube-api-access-ck6xq\") pod \"e1a145f0-821a-4d90-a373-fd65c74c8e57\" (UID: \"e1a145f0-821a-4d90-a373-fd65c74c8e57\") " Nov 02 01:41:31.125240 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.123024 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e1a145f0-821a-4d90-a373-fd65c74c8e57-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1a145f0-821a-4d90-a373-fd65c74c8e57-istio-podinfo\") pod \"e1a145f0-821a-4d90-a373-fd65c74c8e57\" (UID: \"e1a145f0-821a-4d90-a373-fd65c74c8e57\") " Nov 02 01:41:31.125240 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.123065 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1a145f0-821a-4d90-a373-fd65c74c8e57-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1a145f0-821a-4d90-a373-fd65c74c8e57-istio-data\") pod \"e1a145f0-821a-4d90-a373-fd65c74c8e57\" (UID: \"e1a145f0-821a-4d90-a373-fd65c74c8e57\") " Nov 02 01:41:31.125240 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.123136 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1a145f0-821a-4d90-a373-fd65c74c8e57-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1a145f0-821a-4d90-a373-fd65c74c8e57-credential-socket\") pod \"e1a145f0-821a-4d90-a373-fd65c74c8e57\" (UID: \"e1a145f0-821a-4d90-a373-fd65c74c8e57\") " Nov 02 01:41:31.125240 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.123692 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1a145f0-821a-4d90-a373-fd65c74c8e57-workload-certs" pod "e1a145f0-821a-4d90-a373-fd65c74c8e57" (UID: "e1a145f0-821a-4d90-a373-fd65c74c8e57"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:41:31.125240 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.123914 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1a145f0-821a-4d90-a373-fd65c74c8e57-workload-socket" pod "e1a145f0-821a-4d90-a373-fd65c74c8e57" (UID: "e1a145f0-821a-4d90-a373-fd65c74c8e57"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:41:31.125240 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.124834 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1a145f0-821a-4d90-a373-fd65c74c8e57-istiod-ca-cert" pod "e1a145f0-821a-4d90-a373-fd65c74c8e57" (UID: "e1a145f0-821a-4d90-a373-fd65c74c8e57"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:41:31.131806 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.126088 4427 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1a145f0-821a-4d90-a373-fd65c74c8e57-workload-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:31.131806 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.126114 4427 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1a145f0-821a-4d90-a373-fd65c74c8e57-istiod-ca-cert\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:31.131806 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.126129 4427 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1a145f0-821a-4d90-a373-fd65c74c8e57-workload-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:31.131806 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.130633 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1a145f0-821a-4d90-a373-fd65c74c8e57-istio-data" pod "e1a145f0-821a-4d90-a373-fd65c74c8e57" (UID: "e1a145f0-821a-4d90-a373-fd65c74c8e57"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:41:31.131806 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.130885 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1a145f0-821a-4d90-a373-fd65c74c8e57-credential-socket" pod "e1a145f0-821a-4d90-a373-fd65c74c8e57" (UID: "e1a145f0-821a-4d90-a373-fd65c74c8e57"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:41:31.157904 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.155558 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km"] Nov 02 01:41:31.157904 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.156446 4427 cpu_manager.go:414] "containerMap: removing container" podUID="e1a145f0-821a-4d90-a373-fd65c74c8e57" containerName="istio-proxy" Nov 02 01:41:31.157904 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.156460 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1a145f0-821a-4d90-a373-fd65c74c8e57" containerName="istio-proxy" Nov 02 01:41:31.157904 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.156636 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1a145f0-821a-4d90-a373-fd65c74c8e57" containerName="istio-proxy" Nov 02 01:41:31.163589 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.163561 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km" Nov 02 01:41:31.166449 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.165592 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km"] Nov 02 01:41:31.168561 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.168420 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-edef28-openshift-default-dockercfg-bg4zb\"" Nov 02 01:41:31.172841 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.172663 4427 generic.go:356] "Generic (PLEG): container finished" podID="1563efef-2ed3-4347-8399-4c9ca520db4b" containerID="1c33768a9daa419a56994957b1e1cf5c493d0dcd8cad607d9833533acbe3c5ea" exitCode=0 Nov 02 01:41:31.172841 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.172742 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28" event={"ID":"1563efef-2ed3-4347-8399-4c9ca520db4b","Type":"ContainerDied","Data":"1c33768a9daa419a56994957b1e1cf5c493d0dcd8cad607d9833533acbe3c5ea"} Nov 02 01:41:31.174444 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.174287 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1a145f0-821a-4d90-a373-fd65c74c8e57-istio-token" pod "e1a145f0-821a-4d90-a373-fd65c74c8e57" (UID: "e1a145f0-821a-4d90-a373-fd65c74c8e57"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:41:31.175476 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.175438 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1a145f0-821a-4d90-a373-fd65c74c8e57-istio-envoy" pod "e1a145f0-821a-4d90-a373-fd65c74c8e57" (UID: "e1a145f0-821a-4d90-a373-fd65c74c8e57"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:41:31.176706 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.176613 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e1a145f0-821a-4d90-a373-fd65c74c8e57-istio-podinfo" pod "e1a145f0-821a-4d90-a373-fd65c74c8e57" (UID: "e1a145f0-821a-4d90-a373-fd65c74c8e57"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:41:31.176818 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.176748 4427 generic.go:356] "Generic (PLEG): container finished" podID="e1a145f0-821a-4d90-a373-fd65c74c8e57" containerID="97c5be5723ced963fd89d722494b3b29df47f8c81df7a475b1a5a3dac7b8656a" exitCode=0 Nov 02 01:41:31.176818 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.176778 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1a145f0-821a-4d90-a373-fd65c74c8e57-kube-api-access-ck6xq" pod "e1a145f0-821a-4d90-a373-fd65c74c8e57" (UID: "e1a145f0-821a-4d90-a373-fd65c74c8e57"). InnerVolumeSpecName "kube-api-access-ck6xq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:41:31.176932 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.176856 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2" Nov 02 01:41:31.176983 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.176957 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2" event={"ID":"e1a145f0-821a-4d90-a373-fd65c74c8e57","Type":"ContainerDied","Data":"97c5be5723ced963fd89d722494b3b29df47f8c81df7a475b1a5a3dac7b8656a"} Nov 02 01:41:31.177031 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.176991 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2" event={"ID":"e1a145f0-821a-4d90-a373-fd65c74c8e57","Type":"ContainerDied","Data":"7f82efd5152c8ce1f636b88165054cb318804c5f0d1a7f6c8cd20887a2d1e257"} Nov 02 01:41:31.177031 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.177011 4427 scope.go:122] "RemoveContainer" containerID="97c5be5723ced963fd89d722494b3b29df47f8c81df7a475b1a5a3dac7b8656a" Nov 02 01:41:31.181516 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.181446 4427 generic.go:356] "Generic (PLEG): container finished" podID="19218597-2356-4f9d-bf2b-ac4597da575a" containerID="93aae06e287d80ec36542db437424297b58b59e34de95c9a40a02a948dfc96d7" exitCode=0 Nov 02 01:41:31.181632 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.181545 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9" event={"ID":"19218597-2356-4f9d-bf2b-ac4597da575a","Type":"ContainerDied","Data":"93aae06e287d80ec36542db437424297b58b59e34de95c9a40a02a948dfc96d7"} Nov 02 01:41:31.192918 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.192894 4427 scope.go:122] "RemoveContainer" containerID="97c5be5723ced963fd89d722494b3b29df47f8c81df7a475b1a5a3dac7b8656a" Nov 02 01:41:31.193243 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:41:31.193219 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97c5be5723ced963fd89d722494b3b29df47f8c81df7a475b1a5a3dac7b8656a\": container with ID starting with 97c5be5723ced963fd89d722494b3b29df47f8c81df7a475b1a5a3dac7b8656a not found: ID does not exist" containerID="97c5be5723ced963fd89d722494b3b29df47f8c81df7a475b1a5a3dac7b8656a" Nov 02 01:41:31.193318 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.193251 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97c5be5723ced963fd89d722494b3b29df47f8c81df7a475b1a5a3dac7b8656a"} err="failed to get container status \"97c5be5723ced963fd89d722494b3b29df47f8c81df7a475b1a5a3dac7b8656a\": rpc error: code = NotFound desc = could not find container \"97c5be5723ced963fd89d722494b3b29df47f8c81df7a475b1a5a3dac7b8656a\": container with ID starting with 97c5be5723ced963fd89d722494b3b29df47f8c81df7a475b1a5a3dac7b8656a not found: ID does not exist" Nov 02 01:41:31.193318 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.193269 4427 scope.go:122] "RemoveContainer" containerID="6d73244c1f5558e36ec4b736e792c4ed38240123737cf0dd0ccd5bf8a0089be7" Nov 02 01:41:31.210489 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.210458 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2"] Nov 02 01:41:31.218016 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.217991 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-fmwv2"] Nov 02 01:41:31.237253 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.236150 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad15f805-4c58-4931-b0fb-a34a2fb8e4ef-credential-socket\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km\" (UID: \"ad15f805-4c58-4931-b0fb-a34a2fb8e4ef\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km" Nov 02 01:41:31.237253 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.236241 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad15f805-4c58-4931-b0fb-a34a2fb8e4ef-istio-data\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km\" (UID: \"ad15f805-4c58-4931-b0fb-a34a2fb8e4ef\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km" Nov 02 01:41:31.237253 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.236286 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w89ch\" (UniqueName: \"kubernetes.io/projected/ad15f805-4c58-4931-b0fb-a34a2fb8e4ef-kube-api-access-w89ch\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km\" (UID: \"ad15f805-4c58-4931-b0fb-a34a2fb8e4ef\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km" Nov 02 01:41:31.237253 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.236423 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad15f805-4c58-4931-b0fb-a34a2fb8e4ef-workload-socket\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km\" (UID: \"ad15f805-4c58-4931-b0fb-a34a2fb8e4ef\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km" Nov 02 01:41:31.237253 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.236473 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad15f805-4c58-4931-b0fb-a34a2fb8e4ef-istiod-ca-cert\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km\" (UID: \"ad15f805-4c58-4931-b0fb-a34a2fb8e4ef\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km" Nov 02 01:41:31.237253 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.236497 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad15f805-4c58-4931-b0fb-a34a2fb8e4ef-workload-certs\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km\" (UID: \"ad15f805-4c58-4931-b0fb-a34a2fb8e4ef\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km" Nov 02 01:41:31.237253 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.236606 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad15f805-4c58-4931-b0fb-a34a2fb8e4ef-istio-envoy\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km\" (UID: \"ad15f805-4c58-4931-b0fb-a34a2fb8e4ef\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km" Nov 02 01:41:31.237253 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.236677 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad15f805-4c58-4931-b0fb-a34a2fb8e4ef-istio-podinfo\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km\" (UID: \"ad15f805-4c58-4931-b0fb-a34a2fb8e4ef\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km" Nov 02 01:41:31.237253 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.236787 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad15f805-4c58-4931-b0fb-a34a2fb8e4ef-istio-token\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km\" (UID: \"ad15f805-4c58-4931-b0fb-a34a2fb8e4ef\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km" Nov 02 01:41:31.237253 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.236973 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ck6xq\" (UniqueName: \"kubernetes.io/projected/e1a145f0-821a-4d90-a373-fd65c74c8e57-kube-api-access-ck6xq\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:31.237253 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.236988 4427 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1a145f0-821a-4d90-a373-fd65c74c8e57-istio-podinfo\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:31.237253 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.237003 4427 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1a145f0-821a-4d90-a373-fd65c74c8e57-istio-data\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:31.237253 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.237016 4427 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1a145f0-821a-4d90-a373-fd65c74c8e57-credential-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:31.237253 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.237028 4427 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1a145f0-821a-4d90-a373-fd65c74c8e57-istio-token\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:31.237253 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.237039 4427 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1a145f0-821a-4d90-a373-fd65c74c8e57-istio-envoy\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:31.244499 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.242851 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28" Nov 02 01:41:31.282464 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.282416 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp"] Nov 02 01:41:31.283549 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.283522 4427 cpu_manager.go:414] "containerMap: removing container" podUID="1563efef-2ed3-4347-8399-4c9ca520db4b" containerName="istio-proxy" Nov 02 01:41:31.283549 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.283545 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1563efef-2ed3-4347-8399-4c9ca520db4b" containerName="istio-proxy" Nov 02 01:41:31.283812 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.283797 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="1563efef-2ed3-4347-8399-4c9ca520db4b" containerName="istio-proxy" Nov 02 01:41:31.288611 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.288584 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp" Nov 02 01:41:31.302054 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.302023 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp"] Nov 02 01:41:31.338241 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.338173 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1563efef-2ed3-4347-8399-4c9ca520db4b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1563efef-2ed3-4347-8399-4c9ca520db4b-workload-socket\") pod \"1563efef-2ed3-4347-8399-4c9ca520db4b\" (UID: \"1563efef-2ed3-4347-8399-4c9ca520db4b\") " Nov 02 01:41:31.338387 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.338247 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1563efef-2ed3-4347-8399-4c9ca520db4b-kube-api-access-7j5ff\" (UniqueName: \"kubernetes.io/projected/1563efef-2ed3-4347-8399-4c9ca520db4b-kube-api-access-7j5ff\") pod \"1563efef-2ed3-4347-8399-4c9ca520db4b\" (UID: \"1563efef-2ed3-4347-8399-4c9ca520db4b\") " Nov 02 01:41:31.338387 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.338355 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1563efef-2ed3-4347-8399-4c9ca520db4b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1563efef-2ed3-4347-8399-4c9ca520db4b-istiod-ca-cert\") pod \"1563efef-2ed3-4347-8399-4c9ca520db4b\" (UID: \"1563efef-2ed3-4347-8399-4c9ca520db4b\") " Nov 02 01:41:31.338510 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.338490 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1563efef-2ed3-4347-8399-4c9ca520db4b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1563efef-2ed3-4347-8399-4c9ca520db4b-istio-podinfo\") pod \"1563efef-2ed3-4347-8399-4c9ca520db4b\" (UID: \"1563efef-2ed3-4347-8399-4c9ca520db4b\") " Nov 02 01:41:31.338599 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.338516 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1563efef-2ed3-4347-8399-4c9ca520db4b-workload-socket" pod "1563efef-2ed3-4347-8399-4c9ca520db4b" (UID: "1563efef-2ed3-4347-8399-4c9ca520db4b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:41:31.338661 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.338607 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1563efef-2ed3-4347-8399-4c9ca520db4b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1563efef-2ed3-4347-8399-4c9ca520db4b-workload-certs\") pod \"1563efef-2ed3-4347-8399-4c9ca520db4b\" (UID: \"1563efef-2ed3-4347-8399-4c9ca520db4b\") " Nov 02 01:41:31.338661 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.338636 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1563efef-2ed3-4347-8399-4c9ca520db4b-istio-token\" (UniqueName: \"kubernetes.io/projected/1563efef-2ed3-4347-8399-4c9ca520db4b-istio-token\") pod \"1563efef-2ed3-4347-8399-4c9ca520db4b\" (UID: \"1563efef-2ed3-4347-8399-4c9ca520db4b\") " Nov 02 01:41:31.338784 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.338673 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1563efef-2ed3-4347-8399-4c9ca520db4b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1563efef-2ed3-4347-8399-4c9ca520db4b-istio-data\") pod \"1563efef-2ed3-4347-8399-4c9ca520db4b\" (UID: \"1563efef-2ed3-4347-8399-4c9ca520db4b\") " Nov 02 01:41:31.338889 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.338868 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1563efef-2ed3-4347-8399-4c9ca520db4b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1563efef-2ed3-4347-8399-4c9ca520db4b-istio-envoy\") pod \"1563efef-2ed3-4347-8399-4c9ca520db4b\" (UID: \"1563efef-2ed3-4347-8399-4c9ca520db4b\") " Nov 02 01:41:31.338992 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.338936 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1563efef-2ed3-4347-8399-4c9ca520db4b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1563efef-2ed3-4347-8399-4c9ca520db4b-credential-socket\") pod \"1563efef-2ed3-4347-8399-4c9ca520db4b\" (UID: \"1563efef-2ed3-4347-8399-4c9ca520db4b\") " Nov 02 01:41:31.338992 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.338942 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1563efef-2ed3-4347-8399-4c9ca520db4b-workload-certs" pod "1563efef-2ed3-4347-8399-4c9ca520db4b" (UID: "1563efef-2ed3-4347-8399-4c9ca520db4b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:41:31.339154 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.339129 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1563efef-2ed3-4347-8399-4c9ca520db4b-istiod-ca-cert" pod "1563efef-2ed3-4347-8399-4c9ca520db4b" (UID: "1563efef-2ed3-4347-8399-4c9ca520db4b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:41:31.339607 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.339576 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/907ed6a0-8a68-49c1-aa49-916b377a90f5-istiod-ca-cert\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp\" (UID: \"907ed6a0-8a68-49c1-aa49-916b377a90f5\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp" Nov 02 01:41:31.339817 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.339632 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/907ed6a0-8a68-49c1-aa49-916b377a90f5-workload-certs\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp\" (UID: \"907ed6a0-8a68-49c1-aa49-916b377a90f5\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp" Nov 02 01:41:31.339817 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.339717 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad15f805-4c58-4931-b0fb-a34a2fb8e4ef-credential-socket\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km\" (UID: \"ad15f805-4c58-4931-b0fb-a34a2fb8e4ef\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km" Nov 02 01:41:31.339817 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.339764 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/907ed6a0-8a68-49c1-aa49-916b377a90f5-istio-data\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp\" (UID: \"907ed6a0-8a68-49c1-aa49-916b377a90f5\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp" Nov 02 01:41:31.340092 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.339827 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad15f805-4c58-4931-b0fb-a34a2fb8e4ef-istio-data\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km\" (UID: \"ad15f805-4c58-4931-b0fb-a34a2fb8e4ef\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km" Nov 02 01:41:31.340092 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.339874 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w89ch\" (UniqueName: \"kubernetes.io/projected/ad15f805-4c58-4931-b0fb-a34a2fb8e4ef-kube-api-access-w89ch\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km\" (UID: \"ad15f805-4c58-4931-b0fb-a34a2fb8e4ef\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km" Nov 02 01:41:31.340092 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.340009 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad15f805-4c58-4931-b0fb-a34a2fb8e4ef-workload-socket\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km\" (UID: \"ad15f805-4c58-4931-b0fb-a34a2fb8e4ef\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km" Nov 02 01:41:31.340092 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.340061 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/907ed6a0-8a68-49c1-aa49-916b377a90f5-istio-podinfo\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp\" (UID: \"907ed6a0-8a68-49c1-aa49-916b377a90f5\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp" Nov 02 01:41:31.340362 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.340095 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad15f805-4c58-4931-b0fb-a34a2fb8e4ef-istiod-ca-cert\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km\" (UID: \"ad15f805-4c58-4931-b0fb-a34a2fb8e4ef\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km" Nov 02 01:41:31.340362 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.340125 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad15f805-4c58-4931-b0fb-a34a2fb8e4ef-workload-certs\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km\" (UID: \"ad15f805-4c58-4931-b0fb-a34a2fb8e4ef\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km" Nov 02 01:41:31.340362 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.340151 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/907ed6a0-8a68-49c1-aa49-916b377a90f5-credential-socket\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp\" (UID: \"907ed6a0-8a68-49c1-aa49-916b377a90f5\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp" Nov 02 01:41:31.340362 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.340188 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvnq9\" (UniqueName: \"kubernetes.io/projected/907ed6a0-8a68-49c1-aa49-916b377a90f5-kube-api-access-cvnq9\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp\" (UID: \"907ed6a0-8a68-49c1-aa49-916b377a90f5\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp" Nov 02 01:41:31.340362 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.340305 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad15f805-4c58-4931-b0fb-a34a2fb8e4ef-istio-envoy\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km\" (UID: \"ad15f805-4c58-4931-b0fb-a34a2fb8e4ef\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km" Nov 02 01:41:31.340666 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.340398 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/907ed6a0-8a68-49c1-aa49-916b377a90f5-istio-envoy\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp\" (UID: \"907ed6a0-8a68-49c1-aa49-916b377a90f5\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp" Nov 02 01:41:31.340666 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.340428 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/907ed6a0-8a68-49c1-aa49-916b377a90f5-workload-socket\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp\" (UID: \"907ed6a0-8a68-49c1-aa49-916b377a90f5\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp" Nov 02 01:41:31.340666 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.340470 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad15f805-4c58-4931-b0fb-a34a2fb8e4ef-istio-podinfo\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km\" (UID: \"ad15f805-4c58-4931-b0fb-a34a2fb8e4ef\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km" Nov 02 01:41:31.340666 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.340618 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad15f805-4c58-4931-b0fb-a34a2fb8e4ef-istio-token\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km\" (UID: \"ad15f805-4c58-4931-b0fb-a34a2fb8e4ef\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km" Nov 02 01:41:31.340880 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.340678 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/907ed6a0-8a68-49c1-aa49-916b377a90f5-istio-token\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp\" (UID: \"907ed6a0-8a68-49c1-aa49-916b377a90f5\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp" Nov 02 01:41:31.340880 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.340853 4427 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1563efef-2ed3-4347-8399-4c9ca520db4b-istiod-ca-cert\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:31.340880 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.340871 4427 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1563efef-2ed3-4347-8399-4c9ca520db4b-workload-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:31.341105 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.340886 4427 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1563efef-2ed3-4347-8399-4c9ca520db4b-workload-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:31.341742 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.341275 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad15f805-4c58-4931-b0fb-a34a2fb8e4ef-credential-socket\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km\" (UID: \"ad15f805-4c58-4931-b0fb-a34a2fb8e4ef\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km" Nov 02 01:41:31.341742 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.341396 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1563efef-2ed3-4347-8399-4c9ca520db4b-kube-api-access-7j5ff" pod "1563efef-2ed3-4347-8399-4c9ca520db4b" (UID: "1563efef-2ed3-4347-8399-4c9ca520db4b"). InnerVolumeSpecName "kube-api-access-7j5ff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:41:31.341742 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.341422 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1563efef-2ed3-4347-8399-4c9ca520db4b-istio-podinfo" pod "1563efef-2ed3-4347-8399-4c9ca520db4b" (UID: "1563efef-2ed3-4347-8399-4c9ca520db4b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:41:31.341991 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.341758 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad15f805-4c58-4931-b0fb-a34a2fb8e4ef-istio-data\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km\" (UID: \"ad15f805-4c58-4931-b0fb-a34a2fb8e4ef\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km" Nov 02 01:41:31.341991 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.341810 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad15f805-4c58-4931-b0fb-a34a2fb8e4ef-workload-certs\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km\" (UID: \"ad15f805-4c58-4931-b0fb-a34a2fb8e4ef\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km" Nov 02 01:41:31.342278 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.342121 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad15f805-4c58-4931-b0fb-a34a2fb8e4ef-istiod-ca-cert\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km\" (UID: \"ad15f805-4c58-4931-b0fb-a34a2fb8e4ef\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km" Nov 02 01:41:31.342278 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.342153 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad15f805-4c58-4931-b0fb-a34a2fb8e4ef-workload-socket\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km\" (UID: \"ad15f805-4c58-4931-b0fb-a34a2fb8e4ef\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km" Nov 02 01:41:31.342508 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.342473 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1563efef-2ed3-4347-8399-4c9ca520db4b-istio-data" pod "1563efef-2ed3-4347-8399-4c9ca520db4b" (UID: "1563efef-2ed3-4347-8399-4c9ca520db4b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:41:31.342842 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.342814 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1563efef-2ed3-4347-8399-4c9ca520db4b-credential-socket" pod "1563efef-2ed3-4347-8399-4c9ca520db4b" (UID: "1563efef-2ed3-4347-8399-4c9ca520db4b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:41:31.345287 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.345256 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad15f805-4c58-4931-b0fb-a34a2fb8e4ef-istio-podinfo\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km\" (UID: \"ad15f805-4c58-4931-b0fb-a34a2fb8e4ef\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km" Nov 02 01:41:31.346196 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.346151 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1563efef-2ed3-4347-8399-4c9ca520db4b-istio-token" pod "1563efef-2ed3-4347-8399-4c9ca520db4b" (UID: "1563efef-2ed3-4347-8399-4c9ca520db4b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:41:31.346509 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.346476 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1563efef-2ed3-4347-8399-4c9ca520db4b-istio-envoy" pod "1563efef-2ed3-4347-8399-4c9ca520db4b" (UID: "1563efef-2ed3-4347-8399-4c9ca520db4b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:41:31.346991 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.346957 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad15f805-4c58-4931-b0fb-a34a2fb8e4ef-istio-envoy\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km\" (UID: \"ad15f805-4c58-4931-b0fb-a34a2fb8e4ef\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km" Nov 02 01:41:31.354286 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.354223 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad15f805-4c58-4931-b0fb-a34a2fb8e4ef-istio-token\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km\" (UID: \"ad15f805-4c58-4931-b0fb-a34a2fb8e4ef\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km" Nov 02 01:41:31.354836 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.354812 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w89ch\" (UniqueName: \"kubernetes.io/projected/ad15f805-4c58-4931-b0fb-a34a2fb8e4ef-kube-api-access-w89ch\") pod \"e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km\" (UID: \"ad15f805-4c58-4931-b0fb-a34a2fb8e4ef\") " pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km" Nov 02 01:41:31.443086 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.443041 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/907ed6a0-8a68-49c1-aa49-916b377a90f5-istio-data\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp\" (UID: \"907ed6a0-8a68-49c1-aa49-916b377a90f5\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp" Nov 02 01:41:31.443289 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.443267 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/907ed6a0-8a68-49c1-aa49-916b377a90f5-istio-podinfo\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp\" (UID: \"907ed6a0-8a68-49c1-aa49-916b377a90f5\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp" Nov 02 01:41:31.443355 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.443314 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/907ed6a0-8a68-49c1-aa49-916b377a90f5-credential-socket\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp\" (UID: \"907ed6a0-8a68-49c1-aa49-916b377a90f5\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp" Nov 02 01:41:31.443436 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.443348 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvnq9\" (UniqueName: \"kubernetes.io/projected/907ed6a0-8a68-49c1-aa49-916b377a90f5-kube-api-access-cvnq9\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp\" (UID: \"907ed6a0-8a68-49c1-aa49-916b377a90f5\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp" Nov 02 01:41:31.443525 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.443494 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/907ed6a0-8a68-49c1-aa49-916b377a90f5-istio-envoy\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp\" (UID: \"907ed6a0-8a68-49c1-aa49-916b377a90f5\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp" Nov 02 01:41:31.443586 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.443526 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/907ed6a0-8a68-49c1-aa49-916b377a90f5-workload-socket\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp\" (UID: \"907ed6a0-8a68-49c1-aa49-916b377a90f5\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp" Nov 02 01:41:31.443688 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.443668 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/907ed6a0-8a68-49c1-aa49-916b377a90f5-istio-token\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp\" (UID: \"907ed6a0-8a68-49c1-aa49-916b377a90f5\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp" Nov 02 01:41:31.443751 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.443713 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/907ed6a0-8a68-49c1-aa49-916b377a90f5-credential-socket\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp\" (UID: \"907ed6a0-8a68-49c1-aa49-916b377a90f5\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp" Nov 02 01:41:31.443813 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.443784 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/907ed6a0-8a68-49c1-aa49-916b377a90f5-istiod-ca-cert\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp\" (UID: \"907ed6a0-8a68-49c1-aa49-916b377a90f5\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp" Nov 02 01:41:31.443869 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.443820 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/907ed6a0-8a68-49c1-aa49-916b377a90f5-workload-certs\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp\" (UID: \"907ed6a0-8a68-49c1-aa49-916b377a90f5\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp" Nov 02 01:41:31.443992 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.443524 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/907ed6a0-8a68-49c1-aa49-916b377a90f5-istio-data\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp\" (UID: \"907ed6a0-8a68-49c1-aa49-916b377a90f5\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp" Nov 02 01:41:31.444064 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.443972 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j5ff\" (UniqueName: \"kubernetes.io/projected/1563efef-2ed3-4347-8399-4c9ca520db4b-kube-api-access-7j5ff\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:31.444064 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.444031 4427 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1563efef-2ed3-4347-8399-4c9ca520db4b-istio-podinfo\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:31.444064 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.444049 4427 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1563efef-2ed3-4347-8399-4c9ca520db4b-istio-token\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:31.444226 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.444064 4427 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1563efef-2ed3-4347-8399-4c9ca520db4b-istio-data\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:31.444226 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.444080 4427 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1563efef-2ed3-4347-8399-4c9ca520db4b-istio-envoy\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:31.444226 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.444095 4427 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1563efef-2ed3-4347-8399-4c9ca520db4b-credential-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:31.444226 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.444187 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/907ed6a0-8a68-49c1-aa49-916b377a90f5-workload-socket\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp\" (UID: \"907ed6a0-8a68-49c1-aa49-916b377a90f5\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp" Nov 02 01:41:31.444532 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.444509 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/907ed6a0-8a68-49c1-aa49-916b377a90f5-istiod-ca-cert\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp\" (UID: \"907ed6a0-8a68-49c1-aa49-916b377a90f5\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp" Nov 02 01:41:31.444601 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.444563 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/907ed6a0-8a68-49c1-aa49-916b377a90f5-workload-certs\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp\" (UID: \"907ed6a0-8a68-49c1-aa49-916b377a90f5\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp" Nov 02 01:41:31.446762 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.446736 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/907ed6a0-8a68-49c1-aa49-916b377a90f5-istio-envoy\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp\" (UID: \"907ed6a0-8a68-49c1-aa49-916b377a90f5\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp" Nov 02 01:41:31.448631 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.448606 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/907ed6a0-8a68-49c1-aa49-916b377a90f5-istio-podinfo\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp\" (UID: \"907ed6a0-8a68-49c1-aa49-916b377a90f5\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp" Nov 02 01:41:31.458102 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.458080 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/907ed6a0-8a68-49c1-aa49-916b377a90f5-istio-token\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp\" (UID: \"907ed6a0-8a68-49c1-aa49-916b377a90f5\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp" Nov 02 01:41:31.464489 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.464467 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvnq9\" (UniqueName: \"kubernetes.io/projected/907ed6a0-8a68-49c1-aa49-916b377a90f5-kube-api-access-cvnq9\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp\" (UID: \"907ed6a0-8a68-49c1-aa49-916b377a90f5\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp" Nov 02 01:41:31.488746 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.488718 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km" Nov 02 01:41:31.520141 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.520113 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9" Nov 02 01:41:31.541822 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.541781 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1a145f0-821a-4d90-a373-fd65c74c8e57" path="/var/lib/kubelet/pods/e1a145f0-821a-4d90-a373-fd65c74c8e57/volumes" Nov 02 01:41:31.546552 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.545821 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19218597-2356-4f9d-bf2b-ac4597da575a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19218597-2356-4f9d-bf2b-ac4597da575a-istio-envoy\") pod \"19218597-2356-4f9d-bf2b-ac4597da575a\" (UID: \"19218597-2356-4f9d-bf2b-ac4597da575a\") " Nov 02 01:41:31.546552 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.545946 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19218597-2356-4f9d-bf2b-ac4597da575a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19218597-2356-4f9d-bf2b-ac4597da575a-istiod-ca-cert\") pod \"19218597-2356-4f9d-bf2b-ac4597da575a\" (UID: \"19218597-2356-4f9d-bf2b-ac4597da575a\") " Nov 02 01:41:31.546552 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.546131 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19218597-2356-4f9d-bf2b-ac4597da575a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19218597-2356-4f9d-bf2b-ac4597da575a-istio-data\") pod \"19218597-2356-4f9d-bf2b-ac4597da575a\" (UID: \"19218597-2356-4f9d-bf2b-ac4597da575a\") " Nov 02 01:41:31.546552 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.546176 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19218597-2356-4f9d-bf2b-ac4597da575a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19218597-2356-4f9d-bf2b-ac4597da575a-credential-socket\") pod \"19218597-2356-4f9d-bf2b-ac4597da575a\" (UID: \"19218597-2356-4f9d-bf2b-ac4597da575a\") " Nov 02 01:41:31.546552 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.546233 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19218597-2356-4f9d-bf2b-ac4597da575a-kube-api-access-wjgpt\" (UniqueName: \"kubernetes.io/projected/19218597-2356-4f9d-bf2b-ac4597da575a-kube-api-access-wjgpt\") pod \"19218597-2356-4f9d-bf2b-ac4597da575a\" (UID: \"19218597-2356-4f9d-bf2b-ac4597da575a\") " Nov 02 01:41:31.546552 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.546260 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19218597-2356-4f9d-bf2b-ac4597da575a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19218597-2356-4f9d-bf2b-ac4597da575a-workload-socket\") pod \"19218597-2356-4f9d-bf2b-ac4597da575a\" (UID: \"19218597-2356-4f9d-bf2b-ac4597da575a\") " Nov 02 01:41:31.546943 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.546916 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19218597-2356-4f9d-bf2b-ac4597da575a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19218597-2356-4f9d-bf2b-ac4597da575a-istio-podinfo\") pod \"19218597-2356-4f9d-bf2b-ac4597da575a\" (UID: \"19218597-2356-4f9d-bf2b-ac4597da575a\") " Nov 02 01:41:31.547025 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.547009 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19218597-2356-4f9d-bf2b-ac4597da575a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19218597-2356-4f9d-bf2b-ac4597da575a-workload-certs\") pod \"19218597-2356-4f9d-bf2b-ac4597da575a\" (UID: \"19218597-2356-4f9d-bf2b-ac4597da575a\") " Nov 02 01:41:31.547098 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.547067 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19218597-2356-4f9d-bf2b-ac4597da575a-istio-token\" (UniqueName: \"kubernetes.io/projected/19218597-2356-4f9d-bf2b-ac4597da575a-istio-token\") pod \"19218597-2356-4f9d-bf2b-ac4597da575a\" (UID: \"19218597-2356-4f9d-bf2b-ac4597da575a\") " Nov 02 01:41:31.551490 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.551460 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19218597-2356-4f9d-bf2b-ac4597da575a-istio-token" pod "19218597-2356-4f9d-bf2b-ac4597da575a" (UID: "19218597-2356-4f9d-bf2b-ac4597da575a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:41:31.551695 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.551664 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19218597-2356-4f9d-bf2b-ac4597da575a-credential-socket" pod "19218597-2356-4f9d-bf2b-ac4597da575a" (UID: "19218597-2356-4f9d-bf2b-ac4597da575a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:41:31.553234 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.553208 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19218597-2356-4f9d-bf2b-ac4597da575a-workload-certs" pod "19218597-2356-4f9d-bf2b-ac4597da575a" (UID: "19218597-2356-4f9d-bf2b-ac4597da575a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:41:31.553385 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.553273 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19218597-2356-4f9d-bf2b-ac4597da575a-workload-socket" pod "19218597-2356-4f9d-bf2b-ac4597da575a" (UID: "19218597-2356-4f9d-bf2b-ac4597da575a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:41:31.553385 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.553323 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19218597-2356-4f9d-bf2b-ac4597da575a-istio-data" pod "19218597-2356-4f9d-bf2b-ac4597da575a" (UID: "19218597-2356-4f9d-bf2b-ac4597da575a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:41:31.553724 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.553693 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19218597-2356-4f9d-bf2b-ac4597da575a-istiod-ca-cert" pod "19218597-2356-4f9d-bf2b-ac4597da575a" (UID: "19218597-2356-4f9d-bf2b-ac4597da575a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:41:31.560097 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.560052 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19218597-2356-4f9d-bf2b-ac4597da575a-kube-api-access-wjgpt" pod "19218597-2356-4f9d-bf2b-ac4597da575a" (UID: "19218597-2356-4f9d-bf2b-ac4597da575a"). InnerVolumeSpecName "kube-api-access-wjgpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:41:31.564364 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.564342 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm"] Nov 02 01:41:31.565862 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.565836 4427 cpu_manager.go:414] "containerMap: removing container" podUID="19218597-2356-4f9d-bf2b-ac4597da575a" containerName="istio-proxy" Nov 02 01:41:31.565862 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.565862 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19218597-2356-4f9d-bf2b-ac4597da575a" containerName="istio-proxy" Nov 02 01:41:31.566029 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.565888 4427 cpu_manager.go:414] "containerMap: removing container" podUID="19218597-2356-4f9d-bf2b-ac4597da575a" containerName="istio-proxy" Nov 02 01:41:31.566029 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.565896 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19218597-2356-4f9d-bf2b-ac4597da575a" containerName="istio-proxy" Nov 02 01:41:31.566135 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.566054 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="19218597-2356-4f9d-bf2b-ac4597da575a" containerName="istio-proxy" Nov 02 01:41:31.569724 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.569681 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19218597-2356-4f9d-bf2b-ac4597da575a-istio-envoy" pod "19218597-2356-4f9d-bf2b-ac4597da575a" (UID: "19218597-2356-4f9d-bf2b-ac4597da575a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:41:31.571195 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.571111 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19218597-2356-4f9d-bf2b-ac4597da575a-istio-podinfo" pod "19218597-2356-4f9d-bf2b-ac4597da575a" (UID: "19218597-2356-4f9d-bf2b-ac4597da575a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:41:31.586392 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.586349 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm" Nov 02 01:41:31.602881 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.596499 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm"] Nov 02 01:41:31.621461 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.620904 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp" Nov 02 01:41:31.635058 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:41:31.635009 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:41:31.649280 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.649129 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e0dc4b-619a-4759-95af-c288b7257966-credential-socket\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm\" (UID: \"47e0dc4b-619a-4759-95af-c288b7257966\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm" Nov 02 01:41:31.649280 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.649202 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47e0dc4b-619a-4759-95af-c288b7257966-istio-envoy\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm\" (UID: \"47e0dc4b-619a-4759-95af-c288b7257966\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm" Nov 02 01:41:31.649280 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.649251 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scml5\" (UniqueName: \"kubernetes.io/projected/47e0dc4b-619a-4759-95af-c288b7257966-kube-api-access-scml5\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm\" (UID: \"47e0dc4b-619a-4759-95af-c288b7257966\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm" Nov 02 01:41:31.649280 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.649281 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47e0dc4b-619a-4759-95af-c288b7257966-istiod-ca-cert\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm\" (UID: \"47e0dc4b-619a-4759-95af-c288b7257966\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm" Nov 02 01:41:31.649760 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.649318 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47e0dc4b-619a-4759-95af-c288b7257966-workload-certs\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm\" (UID: \"47e0dc4b-619a-4759-95af-c288b7257966\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm" Nov 02 01:41:31.649760 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.649367 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47e0dc4b-619a-4759-95af-c288b7257966-istio-podinfo\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm\" (UID: \"47e0dc4b-619a-4759-95af-c288b7257966\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm" Nov 02 01:41:31.649760 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.649524 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47e0dc4b-619a-4759-95af-c288b7257966-istio-token\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm\" (UID: \"47e0dc4b-619a-4759-95af-c288b7257966\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm" Nov 02 01:41:31.649760 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.649549 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e0dc4b-619a-4759-95af-c288b7257966-workload-socket\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm\" (UID: \"47e0dc4b-619a-4759-95af-c288b7257966\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm" Nov 02 01:41:31.649760 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.649603 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47e0dc4b-619a-4759-95af-c288b7257966-istio-data\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm\" (UID: \"47e0dc4b-619a-4759-95af-c288b7257966\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm" Nov 02 01:41:31.649760 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.649679 4427 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19218597-2356-4f9d-bf2b-ac4597da575a-istio-podinfo\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:31.649760 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.649690 4427 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19218597-2356-4f9d-bf2b-ac4597da575a-workload-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:31.649760 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.649699 4427 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19218597-2356-4f9d-bf2b-ac4597da575a-istio-token\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:31.649760 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.649707 4427 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19218597-2356-4f9d-bf2b-ac4597da575a-istio-envoy\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:31.649760 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.649716 4427 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19218597-2356-4f9d-bf2b-ac4597da575a-istiod-ca-cert\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:31.649760 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.649724 4427 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19218597-2356-4f9d-bf2b-ac4597da575a-istio-data\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:31.649760 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.649732 4427 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19218597-2356-4f9d-bf2b-ac4597da575a-credential-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:31.649760 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.649740 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjgpt\" (UniqueName: \"kubernetes.io/projected/19218597-2356-4f9d-bf2b-ac4597da575a-kube-api-access-wjgpt\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:31.649760 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.649749 4427 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19218597-2356-4f9d-bf2b-ac4597da575a-workload-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:41:31.813162 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.811833 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47e0dc4b-619a-4759-95af-c288b7257966-istio-token\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm\" (UID: \"47e0dc4b-619a-4759-95af-c288b7257966\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm" Nov 02 01:41:31.813162 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.811895 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e0dc4b-619a-4759-95af-c288b7257966-workload-socket\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm\" (UID: \"47e0dc4b-619a-4759-95af-c288b7257966\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm" Nov 02 01:41:31.813162 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.811973 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47e0dc4b-619a-4759-95af-c288b7257966-istio-data\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm\" (UID: \"47e0dc4b-619a-4759-95af-c288b7257966\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm" Nov 02 01:41:31.813162 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.812097 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e0dc4b-619a-4759-95af-c288b7257966-credential-socket\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm\" (UID: \"47e0dc4b-619a-4759-95af-c288b7257966\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm" Nov 02 01:41:31.813162 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.812193 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47e0dc4b-619a-4759-95af-c288b7257966-istio-envoy\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm\" (UID: \"47e0dc4b-619a-4759-95af-c288b7257966\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm" Nov 02 01:41:31.813162 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.812265 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scml5\" (UniqueName: \"kubernetes.io/projected/47e0dc4b-619a-4759-95af-c288b7257966-kube-api-access-scml5\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm\" (UID: \"47e0dc4b-619a-4759-95af-c288b7257966\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm" Nov 02 01:41:31.813162 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.812311 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47e0dc4b-619a-4759-95af-c288b7257966-istiod-ca-cert\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm\" (UID: \"47e0dc4b-619a-4759-95af-c288b7257966\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm" Nov 02 01:41:31.813162 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.812388 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47e0dc4b-619a-4759-95af-c288b7257966-workload-certs\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm\" (UID: \"47e0dc4b-619a-4759-95af-c288b7257966\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm" Nov 02 01:41:31.813162 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.812474 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47e0dc4b-619a-4759-95af-c288b7257966-istio-podinfo\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm\" (UID: \"47e0dc4b-619a-4759-95af-c288b7257966\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm" Nov 02 01:41:31.816536 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.815703 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47e0dc4b-619a-4759-95af-c288b7257966-istio-data\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm\" (UID: \"47e0dc4b-619a-4759-95af-c288b7257966\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm" Nov 02 01:41:31.816536 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.816446 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e0dc4b-619a-4759-95af-c288b7257966-workload-socket\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm\" (UID: \"47e0dc4b-619a-4759-95af-c288b7257966\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm" Nov 02 01:41:31.816934 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.816778 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e0dc4b-619a-4759-95af-c288b7257966-credential-socket\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm\" (UID: \"47e0dc4b-619a-4759-95af-c288b7257966\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm" Nov 02 01:41:31.828538 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.817871 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47e0dc4b-619a-4759-95af-c288b7257966-istiod-ca-cert\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm\" (UID: \"47e0dc4b-619a-4759-95af-c288b7257966\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm" Nov 02 01:41:31.828538 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.820489 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47e0dc4b-619a-4759-95af-c288b7257966-istio-envoy\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm\" (UID: \"47e0dc4b-619a-4759-95af-c288b7257966\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm" Nov 02 01:41:31.828538 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.820642 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47e0dc4b-619a-4759-95af-c288b7257966-istio-podinfo\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm\" (UID: \"47e0dc4b-619a-4759-95af-c288b7257966\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm" Nov 02 01:41:31.828996 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.828951 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47e0dc4b-619a-4759-95af-c288b7257966-workload-certs\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm\" (UID: \"47e0dc4b-619a-4759-95af-c288b7257966\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm" Nov 02 01:41:31.839450 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.837120 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scml5\" (UniqueName: \"kubernetes.io/projected/47e0dc4b-619a-4759-95af-c288b7257966-kube-api-access-scml5\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm\" (UID: \"47e0dc4b-619a-4759-95af-c288b7257966\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm" Nov 02 01:41:31.847193 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.847152 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km"] Nov 02 01:41:31.848154 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:41:31.848119 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad15f805_4c58_4931_b0fb_a34a2fb8e4ef.slice/crio-6ba5362ece87877d67c4539d5a5d526bd5e6c519f32f40fdeacf797d20ba3615 WatchSource:0}: Error finding container 6ba5362ece87877d67c4539d5a5d526bd5e6c519f32f40fdeacf797d20ba3615: Status 404 returned error can't find the container with id 6ba5362ece87877d67c4539d5a5d526bd5e6c519f32f40fdeacf797d20ba3615 Nov 02 01:41:31.853132 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.853080 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47e0dc4b-619a-4759-95af-c288b7257966-istio-token\") pod \"e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm\" (UID: \"47e0dc4b-619a-4759-95af-c288b7257966\") " pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm" Nov 02 01:41:31.856650 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.856609 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:41:31.857092 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.857073 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:41:31.857223 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.857210 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:41:31.911812 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.911751 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm" Nov 02 01:41:31.940676 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.940646 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp"] Nov 02 01:41:31.940676 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:41:31.940659 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod907ed6a0_8a68_49c1_aa49_916b377a90f5.slice/crio-56069eab0114b611733a513bd1f9ba70a98a439a8289fdf91231da5784cc178c WatchSource:0}: Error finding container 56069eab0114b611733a513bd1f9ba70a98a439a8289fdf91231da5784cc178c: Status 404 returned error can't find the container with id 56069eab0114b611733a513bd1f9ba70a98a439a8289fdf91231da5784cc178c Nov 02 01:41:31.944542 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.944503 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:41:31.944624 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.944595 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:41:31.944675 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:31.944637 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:41:32.195906 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:32.195870 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp" event={"ID":"907ed6a0-8a68-49c1-aa49-916b377a90f5","Type":"ContainerStarted","Data":"7cc0cf8471ef328018af21be9b9a9a2880980c333b2561ea4ba40ee7f4ff572c"} Nov 02 01:41:32.196352 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:32.195932 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp" event={"ID":"907ed6a0-8a68-49c1-aa49-916b377a90f5","Type":"ContainerStarted","Data":"56069eab0114b611733a513bd1f9ba70a98a439a8289fdf91231da5784cc178c"} Nov 02 01:41:32.199583 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:32.199536 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28" Nov 02 01:41:32.200332 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:32.199535 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28" event={"ID":"1563efef-2ed3-4347-8399-4c9ca520db4b","Type":"ContainerDied","Data":"96bc815453a44c30c2fb5298ead63b0ed81e0cea378b502f2096411ac31b7fe3"} Nov 02 01:41:32.200468 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:32.200414 4427 scope.go:122] "RemoveContainer" containerID="1c33768a9daa419a56994957b1e1cf5c493d0dcd8cad607d9833533acbe3c5ea" Nov 02 01:41:32.206186 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:32.206153 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km" event={"ID":"ad15f805-4c58-4931-b0fb-a34a2fb8e4ef","Type":"ContainerStarted","Data":"534464f43bd3bb33393303f40960e784a343c9c11ff580c9c8b6ce68012f5c8d"} Nov 02 01:41:32.206303 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:32.206211 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km" event={"ID":"ad15f805-4c58-4931-b0fb-a34a2fb8e4ef","Type":"ContainerStarted","Data":"6ba5362ece87877d67c4539d5a5d526bd5e6c519f32f40fdeacf797d20ba3615"} Nov 02 01:41:32.208030 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:32.207994 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9" Nov 02 01:41:32.208125 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:32.208011 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9" event={"ID":"19218597-2356-4f9d-bf2b-ac4597da575a","Type":"ContainerDied","Data":"bd5f9ba8d5f10d8fe6de66f4d5162454b02f0614b743422ac17970402e08fa66"} Nov 02 01:41:32.232936 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:32.232909 4427 scope.go:122] "RemoveContainer" containerID="93aae06e287d80ec36542db437424297b58b59e34de95c9a40a02a948dfc96d7" Nov 02 01:41:32.235532 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:41:32.235504 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47e0dc4b_619a_4759_95af_c288b7257966.slice/crio-e16a78e89c63f614d898d72ee801ae0caeb2b4bfa2161402814b7a83d9208d0a WatchSource:0}: Error finding container e16a78e89c63f614d898d72ee801ae0caeb2b4bfa2161402814b7a83d9208d0a: Status 404 returned error can't find the container with id e16a78e89c63f614d898d72ee801ae0caeb2b4bfa2161402814b7a83d9208d0a Nov 02 01:41:32.237447 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:32.237366 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm"] Nov 02 01:41:32.238835 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:32.238719 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp" podStartSLOduration=100.238700473 podStartE2EDuration="1m40.238700473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:39:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:41:32.220557169 +0000 UTC m=+1596.841562380" watchObservedRunningTime="2026-11-02 01:41:32.238700473 +0000 UTC m=+1596.859705687" Nov 02 01:41:32.240320 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:32.240295 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:41:32.240483 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:32.240467 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:41:32.240572 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:32.240560 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:41:32.260931 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:32.260876 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km" podStartSLOduration=111.260861831 podStartE2EDuration="1m51.260861831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:39:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:41:32.24922626 +0000 UTC m=+1596.870231472" watchObservedRunningTime="2026-11-02 01:41:32.260861831 +0000 UTC m=+1596.881867042" Nov 02 01:41:32.294539 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:32.294493 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28"] Nov 02 01:41:32.302409 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:32.302363 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-h5j28"] Nov 02 01:41:32.314839 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:32.314808 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9"] Nov 02 01:41:32.323102 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:32.323068 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-q68c9"] Nov 02 01:41:32.490548 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:32.490468 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km" Nov 02 01:41:32.491173 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:32.491111 4427 patch_prober.go:29] interesting pod/e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Nov 02 01:41:32.491293 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:32.491182 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km" podUID="ad15f805-4c58-4931-b0fb-a34a2fb8e4ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Nov 02 01:41:32.623458 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:32.623429 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp" Nov 02 01:41:33.234276 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:33.234220 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm" event={"ID":"47e0dc4b-619a-4759-95af-c288b7257966","Type":"ContainerStarted","Data":"f8c6654ffb98ea4839b23f6da6887cd9200bdc6919b65dd81d2f46d364d9c679"} Nov 02 01:41:33.234841 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:33.234294 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm" event={"ID":"47e0dc4b-619a-4759-95af-c288b7257966","Type":"ContainerStarted","Data":"e16a78e89c63f614d898d72ee801ae0caeb2b4bfa2161402814b7a83d9208d0a"} Nov 02 01:41:33.283262 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:33.283183 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm" podStartSLOduration=101.283158973 podStartE2EDuration="1m41.283158973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:39:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:41:33.26395731 +0000 UTC m=+1597.884962550" watchObservedRunningTime="2026-11-02 01:41:33.283158973 +0000 UTC m=+1597.904164177" Nov 02 01:41:33.507022 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:33.497806 4427 patch_prober.go:29] interesting pod/e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Nov 02 01:41:33.507022 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:33.497870 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km" podUID="ad15f805-4c58-4931-b0fb-a34a2fb8e4ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Nov 02 01:41:33.652655 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:33.651493 4427 patch_prober.go:29] interesting pod/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 02 01:41:33.652655 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:33.651569 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp" podUID="907ed6a0-8a68-49c1-aa49-916b377a90f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 02 01:41:33.655556 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:33.655524 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1563efef-2ed3-4347-8399-4c9ca520db4b" path="/var/lib/kubelet/pods/1563efef-2ed3-4347-8399-4c9ca520db4b/volumes" Nov 02 01:41:33.657061 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:33.657039 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19218597-2356-4f9d-bf2b-ac4597da575a" path="/var/lib/kubelet/pods/19218597-2356-4f9d-bf2b-ac4597da575a/volumes" Nov 02 01:41:33.929398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:33.917303 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm" Nov 02 01:41:33.929398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:33.917451 4427 patch_prober.go:29] interesting pod/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 02 01:41:33.929398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:33.917510 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm" podUID="47e0dc4b-619a-4759-95af-c288b7257966" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 02 01:41:34.492413 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:34.490989 4427 patch_prober.go:29] interesting pod/e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Nov 02 01:41:34.492413 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:34.491062 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km" podUID="ad15f805-4c58-4931-b0fb-a34a2fb8e4ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Nov 02 01:41:34.629397 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:34.623566 4427 patch_prober.go:29] interesting pod/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 02 01:41:34.629397 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:34.623629 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp" podUID="907ed6a0-8a68-49c1-aa49-916b377a90f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 02 01:41:34.922148 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:34.922108 4427 patch_prober.go:29] interesting pod/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 02 01:41:34.922521 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:34.922477 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm" podUID="47e0dc4b-619a-4759-95af-c288b7257966" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 02 01:41:35.502397 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:35.502070 4427 patch_prober.go:29] interesting pod/e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Nov 02 01:41:35.502397 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:35.502130 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km" podUID="ad15f805-4c58-4931-b0fb-a34a2fb8e4ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Nov 02 01:41:35.638576 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:35.637199 4427 patch_prober.go:29] interesting pod/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 02 01:41:35.638576 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:35.637259 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp" podUID="907ed6a0-8a68-49c1-aa49-916b377a90f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 02 01:41:35.913245 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:35.912803 4427 patch_prober.go:29] interesting pod/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 02 01:41:35.913245 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:35.912863 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm" podUID="47e0dc4b-619a-4759-95af-c288b7257966" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 02 01:41:36.496843 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:36.496807 4427 patch_prober.go:29] interesting pod/e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Nov 02 01:41:36.497165 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:36.497111 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km" podUID="ad15f805-4c58-4931-b0fb-a34a2fb8e4ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Nov 02 01:41:36.636874 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:36.636830 4427 patch_prober.go:29] interesting pod/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 02 01:41:36.637531 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:36.637495 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp" podUID="907ed6a0-8a68-49c1-aa49-916b377a90f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 02 01:41:36.927432 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:36.922519 4427 patch_prober.go:29] interesting pod/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 02 01:41:36.927432 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:36.922587 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm" podUID="47e0dc4b-619a-4759-95af-c288b7257966" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 02 01:41:37.496997 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:37.496960 4427 patch_prober.go:29] interesting pod/e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Nov 02 01:41:37.499994 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:37.497245 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km" podUID="ad15f805-4c58-4931-b0fb-a34a2fb8e4ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Nov 02 01:41:37.634401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:37.633365 4427 patch_prober.go:29] interesting pod/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 02 01:41:37.634401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:37.633448 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp" podUID="907ed6a0-8a68-49c1-aa49-916b377a90f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 02 01:41:37.921401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:37.918407 4427 patch_prober.go:29] interesting pod/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 02 01:41:37.921401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:37.918469 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm" podUID="47e0dc4b-619a-4759-95af-c288b7257966" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 02 01:41:38.497401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:38.490811 4427 patch_prober.go:29] interesting pod/e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Nov 02 01:41:38.497401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:38.490886 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km" podUID="ad15f805-4c58-4931-b0fb-a34a2fb8e4ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Nov 02 01:41:38.638404 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:38.634210 4427 patch_prober.go:29] interesting pod/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 02 01:41:38.638404 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:38.634275 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp" podUID="907ed6a0-8a68-49c1-aa49-916b377a90f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 02 01:41:38.912420 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:38.912358 4427 patch_prober.go:29] interesting pod/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 02 01:41:38.912625 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:38.912452 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm" podUID="47e0dc4b-619a-4759-95af-c288b7257966" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 02 01:41:39.497055 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:39.497020 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km" Nov 02 01:41:39.498273 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:39.498142 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km" Nov 02 01:41:39.499093 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:39.499049 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km" Nov 02 01:41:39.631687 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:39.631647 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp" Nov 02 01:41:39.633472 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:39.633446 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp" Nov 02 01:41:39.636599 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:39.636580 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp" Nov 02 01:41:39.955429 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:39.954095 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm" Nov 02 01:41:39.955429 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:39.955133 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm" Nov 02 01:41:39.956240 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:41:39.956218 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm" Nov 02 01:42:01.653065 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:42:01.653028 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:42:19.676293 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:19.676251 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8b3bae-mg6vj"] Nov 02 01:42:19.677800 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:19.677773 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="19218597-2356-4f9d-bf2b-ac4597da575a" containerName="istio-proxy" Nov 02 01:42:19.687357 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:19.687329 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8b3bae-mg6vj"] Nov 02 01:42:19.687549 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:19.687507 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8b3bae-mg6vj" Nov 02 01:42:19.691431 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:19.691190 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 01:42:19.691431 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:19.691224 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\"" Nov 02 01:42:19.744899 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:19.744859 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b771e40-7ddf-432f-bc9b-6444ecd7d743-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8b3bae-mg6vj\" (UID: \"8b771e40-7ddf-432f-bc9b-6444ecd7d743\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8b3bae-mg6vj" Nov 02 01:42:19.745161 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:19.745132 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq5db\" (UniqueName: \"kubernetes.io/projected/8b771e40-7ddf-432f-bc9b-6444ecd7d743-kube-api-access-bq5db\") pod \"maas-api-revoke-keys-e2e-inf-a-8b3bae-mg6vj\" (UID: \"8b771e40-7ddf-432f-bc9b-6444ecd7d743\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8b3bae-mg6vj" Nov 02 01:42:19.745266 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:19.745244 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b771e40-7ddf-432f-bc9b-6444ecd7d743-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8b3bae-mg6vj\" (UID: \"8b771e40-7ddf-432f-bc9b-6444ecd7d743\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8b3bae-mg6vj" Nov 02 01:42:19.847164 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:19.847130 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq5db\" (UniqueName: \"kubernetes.io/projected/8b771e40-7ddf-432f-bc9b-6444ecd7d743-kube-api-access-bq5db\") pod \"maas-api-revoke-keys-e2e-inf-a-8b3bae-mg6vj\" (UID: \"8b771e40-7ddf-432f-bc9b-6444ecd7d743\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8b3bae-mg6vj" Nov 02 01:42:19.847426 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:19.847219 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b771e40-7ddf-432f-bc9b-6444ecd7d743-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8b3bae-mg6vj\" (UID: \"8b771e40-7ddf-432f-bc9b-6444ecd7d743\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8b3bae-mg6vj" Nov 02 01:42:19.847570 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:19.847552 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b771e40-7ddf-432f-bc9b-6444ecd7d743-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8b3bae-mg6vj\" (UID: \"8b771e40-7ddf-432f-bc9b-6444ecd7d743\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8b3bae-mg6vj" Nov 02 01:42:19.848290 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:19.848259 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b771e40-7ddf-432f-bc9b-6444ecd7d743-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8b3bae-mg6vj\" (UID: \"8b771e40-7ddf-432f-bc9b-6444ecd7d743\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8b3bae-mg6vj" Nov 02 01:42:19.850078 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:19.850057 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b771e40-7ddf-432f-bc9b-6444ecd7d743-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8b3bae-mg6vj\" (UID: \"8b771e40-7ddf-432f-bc9b-6444ecd7d743\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8b3bae-mg6vj" Nov 02 01:42:19.859885 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:19.859850 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq5db\" (UniqueName: \"kubernetes.io/projected/8b771e40-7ddf-432f-bc9b-6444ecd7d743-kube-api-access-bq5db\") pod \"maas-api-revoke-keys-e2e-inf-a-8b3bae-mg6vj\" (UID: \"8b771e40-7ddf-432f-bc9b-6444ecd7d743\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8b3bae-mg6vj" Nov 02 01:42:20.006777 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:20.006698 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8b3bae-mg6vj" Nov 02 01:42:20.294873 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:42:20.294836 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b771e40_7ddf_432f_bc9b_6444ecd7d743.slice/crio-7e9b4955ce94d16fe69d9ad6764043f0b29533774760dab67d70ac96572ee615 WatchSource:0}: Error finding container 7e9b4955ce94d16fe69d9ad6764043f0b29533774760dab67d70ac96572ee615: Status 404 returned error can't find the container with id 7e9b4955ce94d16fe69d9ad6764043f0b29533774760dab67d70ac96572ee615 Nov 02 01:42:20.295883 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:20.295825 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8b3bae-mg6vj"] Nov 02 01:42:20.965211 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:20.965122 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8b3bae-mg6vj" event={"ID":"8b771e40-7ddf-432f-bc9b-6444ecd7d743","Type":"ContainerStarted","Data":"e3ed2d7b4272def2ab3a40a0560adbdd93645673103e0cafadc8d3cf736bfd02"} Nov 02 01:42:20.965211 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:20.965206 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8b3bae-mg6vj" event={"ID":"8b771e40-7ddf-432f-bc9b-6444ecd7d743","Type":"ContainerStarted","Data":"7e9b4955ce94d16fe69d9ad6764043f0b29533774760dab67d70ac96572ee615"} Nov 02 01:42:20.987780 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:20.987729 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8b3bae-mg6vj" podStartSLOduration=1.9877150270000001 podStartE2EDuration="1.987715027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:42:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:42:20.985531136 +0000 UTC m=+1645.606536350" watchObservedRunningTime="2026-11-02 01:42:20.987715027 +0000 UTC m=+1645.608720238" Nov 02 01:42:21.979637 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:21.979602 4427 generic.go:356] "Generic (PLEG): container finished" podID="8b771e40-7ddf-432f-bc9b-6444ecd7d743" containerID="e3ed2d7b4272def2ab3a40a0560adbdd93645673103e0cafadc8d3cf736bfd02" exitCode=0 Nov 02 01:42:21.980044 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:21.979685 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8b3bae-mg6vj" event={"ID":"8b771e40-7ddf-432f-bc9b-6444ecd7d743","Type":"ContainerDied","Data":"e3ed2d7b4272def2ab3a40a0560adbdd93645673103e0cafadc8d3cf736bfd02"} Nov 02 01:42:23.175329 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:23.175302 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8b3bae-mg6vj" Nov 02 01:42:23.319048 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:23.318955 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b771e40-7ddf-432f-bc9b-6444ecd7d743-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b771e40-7ddf-432f-bc9b-6444ecd7d743-maas-api-openshift-ca\") pod \"8b771e40-7ddf-432f-bc9b-6444ecd7d743\" (UID: \"8b771e40-7ddf-432f-bc9b-6444ecd7d743\") " Nov 02 01:42:23.319048 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:23.318993 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b771e40-7ddf-432f-bc9b-6444ecd7d743-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b771e40-7ddf-432f-bc9b-6444ecd7d743-maas-api-xks-ca\") pod \"8b771e40-7ddf-432f-bc9b-6444ecd7d743\" (UID: \"8b771e40-7ddf-432f-bc9b-6444ecd7d743\") " Nov 02 01:42:23.319291 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:23.319154 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b771e40-7ddf-432f-bc9b-6444ecd7d743-kube-api-access-bq5db\" (UniqueName: \"kubernetes.io/projected/8b771e40-7ddf-432f-bc9b-6444ecd7d743-kube-api-access-bq5db\") pod \"8b771e40-7ddf-432f-bc9b-6444ecd7d743\" (UID: \"8b771e40-7ddf-432f-bc9b-6444ecd7d743\") " Nov 02 01:42:23.319406 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:23.319363 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b771e40-7ddf-432f-bc9b-6444ecd7d743-maas-api-openshift-ca" pod "8b771e40-7ddf-432f-bc9b-6444ecd7d743" (UID: "8b771e40-7ddf-432f-bc9b-6444ecd7d743"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:42:23.320056 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:23.320037 4427 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b771e40-7ddf-432f-bc9b-6444ecd7d743-maas-api-openshift-ca\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:42:23.321339 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:23.321314 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b771e40-7ddf-432f-bc9b-6444ecd7d743-maas-api-xks-ca" pod "8b771e40-7ddf-432f-bc9b-6444ecd7d743" (UID: "8b771e40-7ddf-432f-bc9b-6444ecd7d743"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:42:23.321452 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:23.321403 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b771e40-7ddf-432f-bc9b-6444ecd7d743-kube-api-access-bq5db" pod "8b771e40-7ddf-432f-bc9b-6444ecd7d743" (UID: "8b771e40-7ddf-432f-bc9b-6444ecd7d743"). InnerVolumeSpecName "kube-api-access-bq5db". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:42:23.422543 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:23.422505 4427 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b771e40-7ddf-432f-bc9b-6444ecd7d743-maas-api-xks-ca\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:42:23.422543 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:23.422534 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq5db\" (UniqueName: \"kubernetes.io/projected/8b771e40-7ddf-432f-bc9b-6444ecd7d743-kube-api-access-bq5db\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:42:23.998207 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:23.998123 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8b3bae-mg6vj" event={"ID":"8b771e40-7ddf-432f-bc9b-6444ecd7d743","Type":"ContainerDied","Data":"7e9b4955ce94d16fe69d9ad6764043f0b29533774760dab67d70ac96572ee615"} Nov 02 01:42:23.998207 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:23.998150 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8b3bae-mg6vj" Nov 02 01:42:23.998207 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:23.998180 4427 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e9b4955ce94d16fe69d9ad6764043f0b29533774760dab67d70ac96572ee615" Nov 02 01:42:28.677098 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:28.677064 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-edef28-59dc7b55f9-vmbqt"] Nov 02 01:42:31.585317 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:42:31.585281 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:42:33.133493 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:33.133460 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bc773aa-g7wls"] Nov 02 01:42:33.151901 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:33.151861 4427 cpu_manager.go:414] "containerMap: removing container" podUID="8b771e40-7ddf-432f-bc9b-6444ecd7d743" containerName="revoke-keys" Nov 02 01:42:33.151901 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:33.151900 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b771e40-7ddf-432f-bc9b-6444ecd7d743" containerName="revoke-keys" Nov 02 01:42:33.153146 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:33.153118 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b771e40-7ddf-432f-bc9b-6444ecd7d743" containerName="revoke-keys" Nov 02 01:42:33.159145 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:33.159106 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bc773aa-g7wls"] Nov 02 01:42:33.159292 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:33.159244 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bc773aa-g7wls" Nov 02 01:42:33.163258 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:33.163235 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 01:42:33.163479 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:33.163338 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\"" Nov 02 01:42:33.228360 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:33.228320 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r57h5\" (UniqueName: \"kubernetes.io/projected/990a3aa9-9b89-4f58-832c-1e257cee5120-kube-api-access-r57h5\") pod \"maas-api-revoke-sub-2bc773aa-g7wls\" (UID: \"990a3aa9-9b89-4f58-832c-1e257cee5120\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bc773aa-g7wls" Nov 02 01:42:33.228653 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:33.228630 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/990a3aa9-9b89-4f58-832c-1e257cee5120-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2bc773aa-g7wls\" (UID: \"990a3aa9-9b89-4f58-832c-1e257cee5120\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bc773aa-g7wls" Nov 02 01:42:33.229044 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:33.229014 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/990a3aa9-9b89-4f58-832c-1e257cee5120-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2bc773aa-g7wls\" (UID: \"990a3aa9-9b89-4f58-832c-1e257cee5120\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bc773aa-g7wls" Nov 02 01:42:33.331256 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:33.331219 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r57h5\" (UniqueName: \"kubernetes.io/projected/990a3aa9-9b89-4f58-832c-1e257cee5120-kube-api-access-r57h5\") pod \"maas-api-revoke-sub-2bc773aa-g7wls\" (UID: \"990a3aa9-9b89-4f58-832c-1e257cee5120\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bc773aa-g7wls" Nov 02 01:42:33.331475 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:33.331283 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/990a3aa9-9b89-4f58-832c-1e257cee5120-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2bc773aa-g7wls\" (UID: \"990a3aa9-9b89-4f58-832c-1e257cee5120\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bc773aa-g7wls" Nov 02 01:42:33.331584 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:33.331493 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/990a3aa9-9b89-4f58-832c-1e257cee5120-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2bc773aa-g7wls\" (UID: \"990a3aa9-9b89-4f58-832c-1e257cee5120\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bc773aa-g7wls" Nov 02 01:42:33.332222 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:33.332195 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/990a3aa9-9b89-4f58-832c-1e257cee5120-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2bc773aa-g7wls\" (UID: \"990a3aa9-9b89-4f58-832c-1e257cee5120\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bc773aa-g7wls" Nov 02 01:42:33.334034 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:33.334012 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/990a3aa9-9b89-4f58-832c-1e257cee5120-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2bc773aa-g7wls\" (UID: \"990a3aa9-9b89-4f58-832c-1e257cee5120\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bc773aa-g7wls" Nov 02 01:42:33.343765 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:33.343740 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r57h5\" (UniqueName: \"kubernetes.io/projected/990a3aa9-9b89-4f58-832c-1e257cee5120-kube-api-access-r57h5\") pod \"maas-api-revoke-sub-2bc773aa-g7wls\" (UID: \"990a3aa9-9b89-4f58-832c-1e257cee5120\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bc773aa-g7wls" Nov 02 01:42:33.482069 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:33.479692 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bc773aa-g7wls" Nov 02 01:42:33.677778 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:33.677735 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edef28-59dc7b55f9-vmbqt" podUID="112b8e02-4d50-426c-a6a5-a6d072abfaed" containerName="payload-pre-processing" containerID="cri-o://419e5518317864f9501a89470d3bbe9ed5d54ecc6d42c605c8ab84063219dca3" gracePeriod=25 Nov 02 01:42:33.814960 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:42:33.814924 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod990a3aa9_9b89_4f58_832c_1e257cee5120.slice/crio-bd6dd1c70152f06b302353ab59003773e18c5e1dde7084bee63d014fadf54ba7 WatchSource:0}: Error finding container bd6dd1c70152f06b302353ab59003773e18c5e1dde7084bee63d014fadf54ba7: Status 404 returned error can't find the container with id bd6dd1c70152f06b302353ab59003773e18c5e1dde7084bee63d014fadf54ba7 Nov 02 01:42:33.817237 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:33.817209 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bc773aa-g7wls"] Nov 02 01:42:33.962558 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:33.962532 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edef28-59dc7b55f9-vmbqt" Nov 02 01:42:34.048887 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:34.048799 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/112b8e02-4d50-426c-a6a5-a6d072abfaed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/112b8e02-4d50-426c-a6a5-a6d072abfaed-plugins-config-volume\") pod \"112b8e02-4d50-426c-a6a5-a6d072abfaed\" (UID: \"112b8e02-4d50-426c-a6a5-a6d072abfaed\") " Nov 02 01:42:34.049058 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:34.048920 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/112b8e02-4d50-426c-a6a5-a6d072abfaed-kube-api-access-qvcjs\" (UniqueName: \"kubernetes.io/projected/112b8e02-4d50-426c-a6a5-a6d072abfaed-kube-api-access-qvcjs\") pod \"112b8e02-4d50-426c-a6a5-a6d072abfaed\" (UID: \"112b8e02-4d50-426c-a6a5-a6d072abfaed\") " Nov 02 01:42:34.049179 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:34.049153 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/112b8e02-4d50-426c-a6a5-a6d072abfaed-plugins-config-volume" pod "112b8e02-4d50-426c-a6a5-a6d072abfaed" (UID: "112b8e02-4d50-426c-a6a5-a6d072abfaed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:42:34.049714 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:34.049692 4427 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/112b8e02-4d50-426c-a6a5-a6d072abfaed-plugins-config-volume\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:42:34.061801 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:34.061768 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/112b8e02-4d50-426c-a6a5-a6d072abfaed-kube-api-access-qvcjs" pod "112b8e02-4d50-426c-a6a5-a6d072abfaed" (UID: "112b8e02-4d50-426c-a6a5-a6d072abfaed"). InnerVolumeSpecName "kube-api-access-qvcjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:42:34.151567 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:34.151532 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvcjs\" (UniqueName: \"kubernetes.io/projected/112b8e02-4d50-426c-a6a5-a6d072abfaed-kube-api-access-qvcjs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:42:34.194214 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:34.194180 4427 generic.go:356] "Generic (PLEG): container finished" podID="112b8e02-4d50-426c-a6a5-a6d072abfaed" containerID="419e5518317864f9501a89470d3bbe9ed5d54ecc6d42c605c8ab84063219dca3" exitCode=0 Nov 02 01:42:34.194441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:34.194233 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edef28-59dc7b55f9-vmbqt" Nov 02 01:42:34.194441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:34.194268 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edef28-59dc7b55f9-vmbqt" event={"ID":"112b8e02-4d50-426c-a6a5-a6d072abfaed","Type":"ContainerDied","Data":"419e5518317864f9501a89470d3bbe9ed5d54ecc6d42c605c8ab84063219dca3"} Nov 02 01:42:34.194441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:34.194321 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edef28-59dc7b55f9-vmbqt" event={"ID":"112b8e02-4d50-426c-a6a5-a6d072abfaed","Type":"ContainerDied","Data":"52de33253a9c11361e37abd7e16aacc012d22cc949d3a1d9fc10b559cfc7033a"} Nov 02 01:42:34.194441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:34.194352 4427 scope.go:122] "RemoveContainer" containerID="419e5518317864f9501a89470d3bbe9ed5d54ecc6d42c605c8ab84063219dca3" Nov 02 01:42:34.196023 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:34.195995 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bc773aa-g7wls" event={"ID":"990a3aa9-9b89-4f58-832c-1e257cee5120","Type":"ContainerStarted","Data":"bd6dd1c70152f06b302353ab59003773e18c5e1dde7084bee63d014fadf54ba7"} Nov 02 01:42:34.208800 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:34.208779 4427 scope.go:122] "RemoveContainer" containerID="419e5518317864f9501a89470d3bbe9ed5d54ecc6d42c605c8ab84063219dca3" Nov 02 01:42:34.209127 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:42:34.209102 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"419e5518317864f9501a89470d3bbe9ed5d54ecc6d42c605c8ab84063219dca3\": container with ID starting with 419e5518317864f9501a89470d3bbe9ed5d54ecc6d42c605c8ab84063219dca3 not found: ID does not exist" containerID="419e5518317864f9501a89470d3bbe9ed5d54ecc6d42c605c8ab84063219dca3" Nov 02 01:42:34.209220 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:34.209134 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"419e5518317864f9501a89470d3bbe9ed5d54ecc6d42c605c8ab84063219dca3"} err="failed to get container status \"419e5518317864f9501a89470d3bbe9ed5d54ecc6d42c605c8ab84063219dca3\": rpc error: code = NotFound desc = could not find container \"419e5518317864f9501a89470d3bbe9ed5d54ecc6d42c605c8ab84063219dca3\": container with ID starting with 419e5518317864f9501a89470d3bbe9ed5d54ecc6d42c605c8ab84063219dca3 not found: ID does not exist" Nov 02 01:42:34.223282 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:34.223251 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-edef28-59dc7b55f9-vmbqt"] Nov 02 01:42:34.230025 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:34.229998 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-edef28-59dc7b55f9-vmbqt"] Nov 02 01:42:35.208997 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:35.208963 4427 generic.go:356] "Generic (PLEG): container finished" podID="990a3aa9-9b89-4f58-832c-1e257cee5120" containerID="8123d1bd02c0f65ae153c107c4879b5c63fe02dce6ac0dce9db7e98d011da67e" exitCode=0 Nov 02 01:42:35.209485 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:35.209089 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bc773aa-g7wls" event={"ID":"990a3aa9-9b89-4f58-832c-1e257cee5120","Type":"ContainerDied","Data":"8123d1bd02c0f65ae153c107c4879b5c63fe02dce6ac0dce9db7e98d011da67e"} Nov 02 01:42:35.489165 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:35.489068 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-edef28-76775f884d-lkrgl"] Nov 02 01:42:35.489630 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:35.489594 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edef28-76775f884d-lkrgl" podUID="2cdaf63a-0f4f-4329-9a71-e6c0a410fec4" containerName="maas-api" containerID="cri-o://0ade303333152e7812ef5638eb7aa8e529330fceede9f00c614d64d7e808ff1f" gracePeriod=30 Nov 02 01:42:35.577302 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:35.577269 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="112b8e02-4d50-426c-a6a5-a6d072abfaed" path="/var/lib/kubelet/pods/112b8e02-4d50-426c-a6a5-a6d072abfaed/volumes" Nov 02 01:42:36.298301 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:36.298262 4427 generic.go:356] "Generic (PLEG): container finished" podID="2cdaf63a-0f4f-4329-9a71-e6c0a410fec4" containerID="0ade303333152e7812ef5638eb7aa8e529330fceede9f00c614d64d7e808ff1f" exitCode=0 Nov 02 01:42:36.298837 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:36.298801 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edef28-76775f884d-lkrgl" event={"ID":"2cdaf63a-0f4f-4329-9a71-e6c0a410fec4","Type":"ContainerDied","Data":"0ade303333152e7812ef5638eb7aa8e529330fceede9f00c614d64d7e808ff1f"} Nov 02 01:42:36.324393 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:36.323417 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edef28-76775f884d-lkrgl" Nov 02 01:42:36.370002 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:36.369965 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd"] Nov 02 01:42:36.371133 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:36.371107 4427 cpu_manager.go:414] "containerMap: removing container" podUID="112b8e02-4d50-426c-a6a5-a6d072abfaed" containerName="payload-pre-processing" Nov 02 01:42:36.371133 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:36.371129 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="112b8e02-4d50-426c-a6a5-a6d072abfaed" containerName="payload-pre-processing" Nov 02 01:42:36.371350 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:36.371187 4427 cpu_manager.go:414] "containerMap: removing container" podUID="2cdaf63a-0f4f-4329-9a71-e6c0a410fec4" containerName="maas-api" Nov 02 01:42:36.371350 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:36.371192 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cdaf63a-0f4f-4329-9a71-e6c0a410fec4" containerName="maas-api" Nov 02 01:42:36.371350 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:36.371306 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cdaf63a-0f4f-4329-9a71-e6c0a410fec4" containerName="maas-api" Nov 02 01:42:36.371350 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:36.371318 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="112b8e02-4d50-426c-a6a5-a6d072abfaed" containerName="payload-pre-processing" Nov 02 01:42:36.375241 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:36.375214 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd" Nov 02 01:42:36.423569 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:36.423532 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd"] Nov 02 01:42:36.523353 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:36.523316 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cdaf63a-0f4f-4329-9a71-e6c0a410fec4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cdaf63a-0f4f-4329-9a71-e6c0a410fec4-openshift-service-ca\") pod \"2cdaf63a-0f4f-4329-9a71-e6c0a410fec4\" (UID: \"2cdaf63a-0f4f-4329-9a71-e6c0a410fec4\") " Nov 02 01:42:36.544562 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:36.534510 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cdaf63a-0f4f-4329-9a71-e6c0a410fec4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cdaf63a-0f4f-4329-9a71-e6c0a410fec4-metrics-certs\") pod \"2cdaf63a-0f4f-4329-9a71-e6c0a410fec4\" (UID: \"2cdaf63a-0f4f-4329-9a71-e6c0a410fec4\") " Nov 02 01:42:36.544899 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:36.544874 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cdaf63a-0f4f-4329-9a71-e6c0a410fec4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cdaf63a-0f4f-4329-9a71-e6c0a410fec4-maas-api-tls\") pod \"2cdaf63a-0f4f-4329-9a71-e6c0a410fec4\" (UID: \"2cdaf63a-0f4f-4329-9a71-e6c0a410fec4\") " Nov 02 01:42:36.545487 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:36.545447 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cdaf63a-0f4f-4329-9a71-e6c0a410fec4-kube-api-access-4ghwq\" (UniqueName: \"kubernetes.io/projected/2cdaf63a-0f4f-4329-9a71-e6c0a410fec4-kube-api-access-4ghwq\") pod \"2cdaf63a-0f4f-4329-9a71-e6c0a410fec4\" (UID: \"2cdaf63a-0f4f-4329-9a71-e6c0a410fec4\") " Nov 02 01:42:36.546072 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:36.546042 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9f64fd6-8cb1-4a34-99f0-828c753a1119-istio-data\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd\" (UID: \"d9f64fd6-8cb1-4a34-99f0-828c753a1119\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd" Nov 02 01:42:36.546246 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:36.546232 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9f64fd6-8cb1-4a34-99f0-828c753a1119-istio-token\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd\" (UID: \"d9f64fd6-8cb1-4a34-99f0-828c753a1119\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd" Nov 02 01:42:36.546367 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:36.546351 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9f64fd6-8cb1-4a34-99f0-828c753a1119-istio-podinfo\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd\" (UID: \"d9f64fd6-8cb1-4a34-99f0-828c753a1119\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd" Nov 02 01:42:36.546530 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:36.546514 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9f64fd6-8cb1-4a34-99f0-828c753a1119-workload-socket\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd\" (UID: \"d9f64fd6-8cb1-4a34-99f0-828c753a1119\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd" Nov 02 01:42:36.546997 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:36.546963 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9f64fd6-8cb1-4a34-99f0-828c753a1119-workload-certs\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd\" (UID: \"d9f64fd6-8cb1-4a34-99f0-828c753a1119\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd" Nov 02 01:42:36.547321 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:36.547300 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9f64fd6-8cb1-4a34-99f0-828c753a1119-istio-envoy\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd\" (UID: \"d9f64fd6-8cb1-4a34-99f0-828c753a1119\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd" Nov 02 01:42:36.547656 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:36.547635 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrfqc\" (UniqueName: \"kubernetes.io/projected/d9f64fd6-8cb1-4a34-99f0-828c753a1119-kube-api-access-xrfqc\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd\" (UID: \"d9f64fd6-8cb1-4a34-99f0-828c753a1119\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd" Nov 02 01:42:36.547876 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:36.547860 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9f64fd6-8cb1-4a34-99f0-828c753a1119-istiod-ca-cert\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd\" (UID: \"d9f64fd6-8cb1-4a34-99f0-828c753a1119\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd" Nov 02 01:42:36.548024 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:36.548010 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9f64fd6-8cb1-4a34-99f0-828c753a1119-credential-socket\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd\" (UID: \"d9f64fd6-8cb1-4a34-99f0-828c753a1119\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd" Nov 02 01:42:36.564750 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:36.564707 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cdaf63a-0f4f-4329-9a71-e6c0a410fec4-openshift-service-ca" pod "2cdaf63a-0f4f-4329-9a71-e6c0a410fec4" (UID: "2cdaf63a-0f4f-4329-9a71-e6c0a410fec4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:42:36.581494 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:36.570976 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cdaf63a-0f4f-4329-9a71-e6c0a410fec4-kube-api-access-4ghwq" pod "2cdaf63a-0f4f-4329-9a71-e6c0a410fec4" (UID: "2cdaf63a-0f4f-4329-9a71-e6c0a410fec4"). InnerVolumeSpecName "kube-api-access-4ghwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:42:36.586508 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:36.583543 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cdaf63a-0f4f-4329-9a71-e6c0a410fec4-maas-api-tls" pod "2cdaf63a-0f4f-4329-9a71-e6c0a410fec4" (UID: "2cdaf63a-0f4f-4329-9a71-e6c0a410fec4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:42:36.588532 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:36.587012 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cdaf63a-0f4f-4329-9a71-e6c0a410fec4-metrics-certs" pod "2cdaf63a-0f4f-4329-9a71-e6c0a410fec4" (UID: "2cdaf63a-0f4f-4329-9a71-e6c0a410fec4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:42:36.627282 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:36.627254 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bc773aa-g7wls" Nov 02 01:42:36.650086 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:36.650047 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/990a3aa9-9b89-4f58-832c-1e257cee5120-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/990a3aa9-9b89-4f58-832c-1e257cee5120-maas-api-openshift-ca\") pod \"990a3aa9-9b89-4f58-832c-1e257cee5120\" (UID: \"990a3aa9-9b89-4f58-832c-1e257cee5120\") " Nov 02 01:42:36.650281 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:36.650190 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/990a3aa9-9b89-4f58-832c-1e257cee5120-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/990a3aa9-9b89-4f58-832c-1e257cee5120-maas-api-xks-ca\") pod \"990a3aa9-9b89-4f58-832c-1e257cee5120\" (UID: \"990a3aa9-9b89-4f58-832c-1e257cee5120\") " Nov 02 01:42:36.650397 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:36.650349 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/990a3aa9-9b89-4f58-832c-1e257cee5120-kube-api-access-r57h5\" (UniqueName: \"kubernetes.io/projected/990a3aa9-9b89-4f58-832c-1e257cee5120-kube-api-access-r57h5\") pod \"990a3aa9-9b89-4f58-832c-1e257cee5120\" (UID: \"990a3aa9-9b89-4f58-832c-1e257cee5120\") " Nov 02 01:42:36.650965 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:36.650878 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9f64fd6-8cb1-4a34-99f0-828c753a1119-istio-data\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd\" (UID: \"d9f64fd6-8cb1-4a34-99f0-828c753a1119\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd" Nov 02 01:42:36.650965 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:36.650925 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/990a3aa9-9b89-4f58-832c-1e257cee5120-maas-api-openshift-ca" pod "990a3aa9-9b89-4f58-832c-1e257cee5120" (UID: "990a3aa9-9b89-4f58-832c-1e257cee5120"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:42:36.650965 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:36.650932 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9f64fd6-8cb1-4a34-99f0-828c753a1119-istio-token\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd\" (UID: \"d9f64fd6-8cb1-4a34-99f0-828c753a1119\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd" Nov 02 01:42:36.651681 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:36.651274 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9f64fd6-8cb1-4a34-99f0-828c753a1119-istio-podinfo\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd\" (UID: \"d9f64fd6-8cb1-4a34-99f0-828c753a1119\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd" Nov 02 01:42:36.651681 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:36.651336 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9f64fd6-8cb1-4a34-99f0-828c753a1119-workload-socket\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd\" (UID: \"d9f64fd6-8cb1-4a34-99f0-828c753a1119\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd" Nov 02 01:42:36.651681 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:36.651548 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9f64fd6-8cb1-4a34-99f0-828c753a1119-workload-certs\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd\" (UID: \"d9f64fd6-8cb1-4a34-99f0-828c753a1119\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd" Nov 02 01:42:36.651924 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:36.651747 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9f64fd6-8cb1-4a34-99f0-828c753a1119-istio-envoy\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd\" (UID: \"d9f64fd6-8cb1-4a34-99f0-828c753a1119\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd" Nov 02 01:42:36.651924 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:36.651783 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9f64fd6-8cb1-4a34-99f0-828c753a1119-workload-socket\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd\" (UID: \"d9f64fd6-8cb1-4a34-99f0-828c753a1119\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd" Nov 02 01:42:36.652034 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:36.651959 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrfqc\" (UniqueName: \"kubernetes.io/projected/d9f64fd6-8cb1-4a34-99f0-828c753a1119-kube-api-access-xrfqc\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd\" (UID: \"d9f64fd6-8cb1-4a34-99f0-828c753a1119\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd" Nov 02 01:42:36.652163 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:36.652137 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9f64fd6-8cb1-4a34-99f0-828c753a1119-istiod-ca-cert\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd\" (UID: \"d9f64fd6-8cb1-4a34-99f0-828c753a1119\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd" Nov 02 01:42:36.652351 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:36.652223 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9f64fd6-8cb1-4a34-99f0-828c753a1119-credential-socket\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd\" (UID: \"d9f64fd6-8cb1-4a34-99f0-828c753a1119\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd" Nov 02 01:42:36.652734 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:36.651567 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9f64fd6-8cb1-4a34-99f0-828c753a1119-istio-data\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd\" (UID: \"d9f64fd6-8cb1-4a34-99f0-828c753a1119\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd" Nov 02 01:42:36.652853 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:36.652771 4427 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cdaf63a-0f4f-4329-9a71-e6c0a410fec4-openshift-service-ca\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:42:36.653131 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:36.653107 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9f64fd6-8cb1-4a34-99f0-828c753a1119-workload-certs\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd\" (UID: \"d9f64fd6-8cb1-4a34-99f0-828c753a1119\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd" Nov 02 01:42:36.653335 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:36.653306 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9f64fd6-8cb1-4a34-99f0-828c753a1119-credential-socket\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd\" (UID: \"d9f64fd6-8cb1-4a34-99f0-828c753a1119\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd" Nov 02 01:42:36.653335 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:36.653297 4427 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cdaf63a-0f4f-4329-9a71-e6c0a410fec4-metrics-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:42:36.653543 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:36.653431 4427 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cdaf63a-0f4f-4329-9a71-e6c0a410fec4-maas-api-tls\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:42:36.653543 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:36.653451 4427 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/990a3aa9-9b89-4f58-832c-1e257cee5120-maas-api-openshift-ca\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:42:36.653543 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:36.653467 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ghwq\" (UniqueName: \"kubernetes.io/projected/2cdaf63a-0f4f-4329-9a71-e6c0a410fec4-kube-api-access-4ghwq\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:42:36.654199 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:36.654176 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9f64fd6-8cb1-4a34-99f0-828c753a1119-istiod-ca-cert\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd\" (UID: \"d9f64fd6-8cb1-4a34-99f0-828c753a1119\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd" Nov 02 01:42:36.655976 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:36.655954 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/990a3aa9-9b89-4f58-832c-1e257cee5120-maas-api-xks-ca" pod "990a3aa9-9b89-4f58-832c-1e257cee5120" (UID: "990a3aa9-9b89-4f58-832c-1e257cee5120"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:42:36.656315 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:36.656293 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9f64fd6-8cb1-4a34-99f0-828c753a1119-istio-podinfo\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd\" (UID: \"d9f64fd6-8cb1-4a34-99f0-828c753a1119\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd" Nov 02 01:42:36.657151 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:36.657123 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/990a3aa9-9b89-4f58-832c-1e257cee5120-kube-api-access-r57h5" pod "990a3aa9-9b89-4f58-832c-1e257cee5120" (UID: "990a3aa9-9b89-4f58-832c-1e257cee5120"). InnerVolumeSpecName "kube-api-access-r57h5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:42:36.657430 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:36.657410 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9f64fd6-8cb1-4a34-99f0-828c753a1119-istio-envoy\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd\" (UID: \"d9f64fd6-8cb1-4a34-99f0-828c753a1119\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd" Nov 02 01:42:36.664294 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:36.664271 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9f64fd6-8cb1-4a34-99f0-828c753a1119-istio-token\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd\" (UID: \"d9f64fd6-8cb1-4a34-99f0-828c753a1119\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd" Nov 02 01:42:36.672063 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:36.672037 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrfqc\" (UniqueName: \"kubernetes.io/projected/d9f64fd6-8cb1-4a34-99f0-828c753a1119-kube-api-access-xrfqc\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd\" (UID: \"d9f64fd6-8cb1-4a34-99f0-828c753a1119\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd" Nov 02 01:42:36.738563 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:36.738533 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd" Nov 02 01:42:36.755311 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:36.755285 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r57h5\" (UniqueName: \"kubernetes.io/projected/990a3aa9-9b89-4f58-832c-1e257cee5120-kube-api-access-r57h5\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:42:36.755311 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:36.755312 4427 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/990a3aa9-9b89-4f58-832c-1e257cee5120-maas-api-xks-ca\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:42:37.038949 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:37.038913 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd"] Nov 02 01:42:37.039562 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:42:37.039530 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9f64fd6_8cb1_4a34_99f0_828c753a1119.slice/crio-75748979247eec2a3f049298fc50eff90c26aa01ec9939714047e889a64140e8 WatchSource:0}: Error finding container 75748979247eec2a3f049298fc50eff90c26aa01ec9939714047e889a64140e8: Status 404 returned error can't find the container with id 75748979247eec2a3f049298fc50eff90c26aa01ec9939714047e889a64140e8 Nov 02 01:42:37.042947 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:37.042913 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:42:37.043052 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:37.042983 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:42:37.043052 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:37.043015 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:42:37.318080 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:37.317975 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edef28-76775f884d-lkrgl" event={"ID":"2cdaf63a-0f4f-4329-9a71-e6c0a410fec4","Type":"ContainerDied","Data":"58559291f5946f3df5f72ccc90cc0357f669138771e1fc003f52924ffc77e1dd"} Nov 02 01:42:37.318628 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:37.318093 4427 scope.go:122] "RemoveContainer" containerID="0ade303333152e7812ef5638eb7aa8e529330fceede9f00c614d64d7e808ff1f" Nov 02 01:42:37.319418 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:37.319398 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edef28-76775f884d-lkrgl" Nov 02 01:42:37.321470 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:37.321195 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bc773aa-g7wls" event={"ID":"990a3aa9-9b89-4f58-832c-1e257cee5120","Type":"ContainerDied","Data":"bd6dd1c70152f06b302353ab59003773e18c5e1dde7084bee63d014fadf54ba7"} Nov 02 01:42:37.321470 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:37.321258 4427 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd6dd1c70152f06b302353ab59003773e18c5e1dde7084bee63d014fadf54ba7" Nov 02 01:42:37.321470 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:37.321452 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bc773aa-g7wls" Nov 02 01:42:37.326126 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:37.325017 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd" event={"ID":"d9f64fd6-8cb1-4a34-99f0-828c753a1119","Type":"ContainerStarted","Data":"ba3e2481997f8ddb323e8b6862e4cf4a4bd794e8f35a8e774e0aae11265a8a5a"} Nov 02 01:42:37.326126 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:37.325065 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd" event={"ID":"d9f64fd6-8cb1-4a34-99f0-828c753a1119","Type":"ContainerStarted","Data":"75748979247eec2a3f049298fc50eff90c26aa01ec9939714047e889a64140e8"} Nov 02 01:42:37.359608 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:37.359548 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd" podStartSLOduration=164.359529087 podStartE2EDuration="2m44.359529087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:39:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:42:37.351751294 +0000 UTC m=+1661.972756507" watchObservedRunningTime="2026-11-02 01:42:37.359529087 +0000 UTC m=+1661.980534297" Nov 02 01:42:37.437762 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:37.427463 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-edef28-76775f884d-lkrgl"] Nov 02 01:42:37.492399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:37.491391 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-edef28-76775f884d-lkrgl"] Nov 02 01:42:37.724403 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:37.721400 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cdaf63a-0f4f-4329-9a71-e6c0a410fec4" path="/var/lib/kubelet/pods/2cdaf63a-0f4f-4329-9a71-e6c0a410fec4/volumes" Nov 02 01:42:37.742406 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:37.741456 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd" Nov 02 01:42:38.031873 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:38.031782 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8b3bae-mg6vj"] Nov 02 01:42:38.038198 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:38.038169 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8b3bae-mg6vj"] Nov 02 01:42:38.740902 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:38.740859 4427 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 02 01:42:38.741425 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:38.740920 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd" podUID="d9f64fd6-8cb1-4a34-99f0-828c753a1119" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 02 01:42:39.180398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:39.179472 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edef28-76775f884d-lkrgl" podUID="2cdaf63a-0f4f-4329-9a71-e6c0a410fec4" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.129:8443/health\": context deadline exceeded" Nov 02 01:42:39.538605 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:39.538526 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b771e40-7ddf-432f-bc9b-6444ecd7d743" path="/var/lib/kubelet/pods/8b771e40-7ddf-432f-bc9b-6444ecd7d743/volumes" Nov 02 01:42:39.739751 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:39.739717 4427 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 02 01:42:39.739938 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:39.739768 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd" podUID="d9f64fd6-8cb1-4a34-99f0-828c753a1119" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 02 01:42:40.739069 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:40.739032 4427 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 02 01:42:40.739498 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:40.739089 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd" podUID="d9f64fd6-8cb1-4a34-99f0-828c753a1119" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 02 01:42:41.739546 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:41.739509 4427 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 02 01:42:41.739915 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:41.739576 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd" podUID="d9f64fd6-8cb1-4a34-99f0-828c753a1119" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 02 01:42:42.739287 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:42.739253 4427 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 02 01:42:42.739512 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:42.739308 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd" podUID="d9f64fd6-8cb1-4a34-99f0-828c753a1119" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 02 01:42:43.749225 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:43.743530 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-0bbeda/test-model-0bbeda-kserve-5cb56fbff9-lqp9j"] Nov 02 01:42:43.791526 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:43.791491 4427 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 02 01:42:43.791805 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:43.791771 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd" podUID="d9f64fd6-8cb1-4a34-99f0-828c753a1119" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 02 01:42:44.151993 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:44.151948 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km"] Nov 02 01:42:44.152503 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:44.152470 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km" podUID="ad15f805-4c58-4931-b0fb-a34a2fb8e4ef" containerName="istio-proxy" containerID="cri-o://534464f43bd3bb33393303f40960e784a343c9c11ff580c9c8b6ce68012f5c8d" gracePeriod=30 Nov 02 01:42:44.827398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:44.815594 4427 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 02 01:42:44.827398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:44.815679 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd" podUID="d9f64fd6-8cb1-4a34-99f0-828c753a1119" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 02 01:42:44.827398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:44.816478 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" podUID="cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 02 01:42:44.884785 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:44.866955 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" podUID="f67c954a-ce5b-41d9-8d99-7d126bd89d59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:42:44.886731 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:44.885530 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" podUID="eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Nov 02 01:42:44.903518 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:44.894994 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" podUID="353a8a41-21ee-4470-861c-b53833beb106" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:42:45.090737 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:45.083676 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" podUID="d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Nov 02 01:42:45.108955 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:45.108879 4427 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" podUID="7085c0ab-7ee3-4665-abad-2977c25f68ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": context deadline exceeded" Nov 02 01:42:45.327537 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:45.298675 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-12e7cb/test-model-12e7cb-kserve-5b5d7468c9-xkkqs" podUID="21ee65a5-81e6-4736-91ac-6399ba34947e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.131:8000/ready\": context deadline exceeded" Nov 02 01:42:45.993006 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:45.992974 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd" Nov 02 01:42:45.993612 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:45.993438 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd" Nov 02 01:42:45.994581 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:45.994562 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd" Nov 02 01:42:46.567486 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:46.567449 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h"] Nov 02 01:42:46.567740 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:46.567710 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h" podUID="95975531-e9c3-4fcb-9f93-ddab47bb62ba" containerName="istio-proxy" containerID="cri-o://67f8e1110ac77d3bcdbc8748e701a2e277e503141cc9103464ad79bec919c73c" gracePeriod=30 Nov 02 01:42:49.425513 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:49.425483 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km" Nov 02 01:42:49.465332 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:49.465239 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm"] Nov 02 01:42:49.466786 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:49.466754 4427 cpu_manager.go:414] "containerMap: removing container" podUID="990a3aa9-9b89-4f58-832c-1e257cee5120" containerName="revoke-keys" Nov 02 01:42:49.466786 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:49.466783 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="990a3aa9-9b89-4f58-832c-1e257cee5120" containerName="revoke-keys" Nov 02 01:42:49.467006 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:49.466823 4427 cpu_manager.go:414] "containerMap: removing container" podUID="ad15f805-4c58-4931-b0fb-a34a2fb8e4ef" containerName="istio-proxy" Nov 02 01:42:49.467006 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:49.466831 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad15f805-4c58-4931-b0fb-a34a2fb8e4ef" containerName="istio-proxy" Nov 02 01:42:49.467125 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:49.467076 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="990a3aa9-9b89-4f58-832c-1e257cee5120" containerName="revoke-keys" Nov 02 01:42:49.467125 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:49.467096 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad15f805-4c58-4931-b0fb-a34a2fb8e4ef" containerName="istio-proxy" Nov 02 01:42:49.472267 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:49.471415 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm" Nov 02 01:42:49.497032 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:49.496995 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm"] Nov 02 01:42:49.603387 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:49.603341 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad15f805-4c58-4931-b0fb-a34a2fb8e4ef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad15f805-4c58-4931-b0fb-a34a2fb8e4ef-workload-certs\") pod \"ad15f805-4c58-4931-b0fb-a34a2fb8e4ef\" (UID: \"ad15f805-4c58-4931-b0fb-a34a2fb8e4ef\") " Nov 02 01:42:49.603576 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:49.603439 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad15f805-4c58-4931-b0fb-a34a2fb8e4ef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad15f805-4c58-4931-b0fb-a34a2fb8e4ef-workload-socket\") pod \"ad15f805-4c58-4931-b0fb-a34a2fb8e4ef\" (UID: \"ad15f805-4c58-4931-b0fb-a34a2fb8e4ef\") " Nov 02 01:42:49.603576 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:49.603514 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad15f805-4c58-4931-b0fb-a34a2fb8e4ef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad15f805-4c58-4931-b0fb-a34a2fb8e4ef-istio-envoy\") pod \"ad15f805-4c58-4931-b0fb-a34a2fb8e4ef\" (UID: \"ad15f805-4c58-4931-b0fb-a34a2fb8e4ef\") " Nov 02 01:42:49.603761 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:49.603584 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad15f805-4c58-4931-b0fb-a34a2fb8e4ef-istio-token\" (UniqueName: \"kubernetes.io/projected/ad15f805-4c58-4931-b0fb-a34a2fb8e4ef-istio-token\") pod \"ad15f805-4c58-4931-b0fb-a34a2fb8e4ef\" (UID: \"ad15f805-4c58-4931-b0fb-a34a2fb8e4ef\") " Nov 02 01:42:49.603761 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:49.603642 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad15f805-4c58-4931-b0fb-a34a2fb8e4ef-workload-certs" pod "ad15f805-4c58-4931-b0fb-a34a2fb8e4ef" (UID: "ad15f805-4c58-4931-b0fb-a34a2fb8e4ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:42:49.603761 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:49.603696 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad15f805-4c58-4931-b0fb-a34a2fb8e4ef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad15f805-4c58-4931-b0fb-a34a2fb8e4ef-istio-data\") pod \"ad15f805-4c58-4931-b0fb-a34a2fb8e4ef\" (UID: \"ad15f805-4c58-4931-b0fb-a34a2fb8e4ef\") " Nov 02 01:42:49.603930 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:49.603795 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad15f805-4c58-4931-b0fb-a34a2fb8e4ef-workload-socket" pod "ad15f805-4c58-4931-b0fb-a34a2fb8e4ef" (UID: "ad15f805-4c58-4931-b0fb-a34a2fb8e4ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:42:49.603990 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:49.603933 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad15f805-4c58-4931-b0fb-a34a2fb8e4ef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad15f805-4c58-4931-b0fb-a34a2fb8e4ef-credential-socket\") pod \"ad15f805-4c58-4931-b0fb-a34a2fb8e4ef\" (UID: \"ad15f805-4c58-4931-b0fb-a34a2fb8e4ef\") " Nov 02 01:42:49.603990 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:49.603972 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad15f805-4c58-4931-b0fb-a34a2fb8e4ef-istio-data" pod "ad15f805-4c58-4931-b0fb-a34a2fb8e4ef" (UID: "ad15f805-4c58-4931-b0fb-a34a2fb8e4ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:42:49.604071 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:49.604019 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad15f805-4c58-4931-b0fb-a34a2fb8e4ef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad15f805-4c58-4931-b0fb-a34a2fb8e4ef-istio-podinfo\") pod \"ad15f805-4c58-4931-b0fb-a34a2fb8e4ef\" (UID: \"ad15f805-4c58-4931-b0fb-a34a2fb8e4ef\") " Nov 02 01:42:49.604134 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:49.604095 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad15f805-4c58-4931-b0fb-a34a2fb8e4ef-kube-api-access-w89ch\" (UniqueName: \"kubernetes.io/projected/ad15f805-4c58-4931-b0fb-a34a2fb8e4ef-kube-api-access-w89ch\") pod \"ad15f805-4c58-4931-b0fb-a34a2fb8e4ef\" (UID: \"ad15f805-4c58-4931-b0fb-a34a2fb8e4ef\") " Nov 02 01:42:49.604134 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:49.604126 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad15f805-4c58-4931-b0fb-a34a2fb8e4ef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad15f805-4c58-4931-b0fb-a34a2fb8e4ef-istiod-ca-cert\") pod \"ad15f805-4c58-4931-b0fb-a34a2fb8e4ef\" (UID: \"ad15f805-4c58-4931-b0fb-a34a2fb8e4ef\") " Nov 02 01:42:49.604242 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:49.604227 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad15f805-4c58-4931-b0fb-a34a2fb8e4ef-credential-socket" pod "ad15f805-4c58-4931-b0fb-a34a2fb8e4ef" (UID: "ad15f805-4c58-4931-b0fb-a34a2fb8e4ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:42:49.604581 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:49.604550 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/136eabea-7a63-4d89-bcc1-0e3d6c8810cf-istio-podinfo\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm\" (UID: \"136eabea-7a63-4d89-bcc1-0e3d6c8810cf\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm" Nov 02 01:42:49.604691 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:49.604611 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/136eabea-7a63-4d89-bcc1-0e3d6c8810cf-istio-token\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm\" (UID: \"136eabea-7a63-4d89-bcc1-0e3d6c8810cf\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm" Nov 02 01:42:49.604756 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:49.604685 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad15f805-4c58-4931-b0fb-a34a2fb8e4ef-istiod-ca-cert" pod "ad15f805-4c58-4931-b0fb-a34a2fb8e4ef" (UID: "ad15f805-4c58-4931-b0fb-a34a2fb8e4ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:42:49.604756 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:49.604693 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/136eabea-7a63-4d89-bcc1-0e3d6c8810cf-workload-certs\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm\" (UID: \"136eabea-7a63-4d89-bcc1-0e3d6c8810cf\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm" Nov 02 01:42:49.604939 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:49.604915 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/136eabea-7a63-4d89-bcc1-0e3d6c8810cf-workload-socket\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm\" (UID: \"136eabea-7a63-4d89-bcc1-0e3d6c8810cf\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm" Nov 02 01:42:49.605217 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:49.604973 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkhcn\" (UniqueName: \"kubernetes.io/projected/136eabea-7a63-4d89-bcc1-0e3d6c8810cf-kube-api-access-tkhcn\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm\" (UID: \"136eabea-7a63-4d89-bcc1-0e3d6c8810cf\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm" Nov 02 01:42:49.605312 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:49.605256 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/136eabea-7a63-4d89-bcc1-0e3d6c8810cf-istio-envoy\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm\" (UID: \"136eabea-7a63-4d89-bcc1-0e3d6c8810cf\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm" Nov 02 01:42:49.605418 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:49.605332 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/136eabea-7a63-4d89-bcc1-0e3d6c8810cf-istio-data\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm\" (UID: \"136eabea-7a63-4d89-bcc1-0e3d6c8810cf\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm" Nov 02 01:42:49.605500 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:49.605474 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/136eabea-7a63-4d89-bcc1-0e3d6c8810cf-istiod-ca-cert\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm\" (UID: \"136eabea-7a63-4d89-bcc1-0e3d6c8810cf\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm" Nov 02 01:42:49.605594 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:49.605534 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/136eabea-7a63-4d89-bcc1-0e3d6c8810cf-credential-socket\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm\" (UID: \"136eabea-7a63-4d89-bcc1-0e3d6c8810cf\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm" Nov 02 01:42:49.605771 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:49.605751 4427 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad15f805-4c58-4931-b0fb-a34a2fb8e4ef-credential-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:42:49.605853 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:49.605779 4427 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad15f805-4c58-4931-b0fb-a34a2fb8e4ef-istiod-ca-cert\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:42:49.605853 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:49.605795 4427 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad15f805-4c58-4931-b0fb-a34a2fb8e4ef-workload-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:42:49.606004 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:49.605892 4427 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad15f805-4c58-4931-b0fb-a34a2fb8e4ef-workload-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:42:49.606004 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:49.605920 4427 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad15f805-4c58-4931-b0fb-a34a2fb8e4ef-istio-data\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:42:49.607553 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:49.607522 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad15f805-4c58-4931-b0fb-a34a2fb8e4ef-kube-api-access-w89ch" pod "ad15f805-4c58-4931-b0fb-a34a2fb8e4ef" (UID: "ad15f805-4c58-4931-b0fb-a34a2fb8e4ef"). InnerVolumeSpecName "kube-api-access-w89ch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:42:49.609678 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:49.609647 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad15f805-4c58-4931-b0fb-a34a2fb8e4ef-istio-podinfo" pod "ad15f805-4c58-4931-b0fb-a34a2fb8e4ef" (UID: "ad15f805-4c58-4931-b0fb-a34a2fb8e4ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:42:49.609819 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:49.609778 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad15f805-4c58-4931-b0fb-a34a2fb8e4ef-istio-token" pod "ad15f805-4c58-4931-b0fb-a34a2fb8e4ef" (UID: "ad15f805-4c58-4931-b0fb-a34a2fb8e4ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:42:49.611273 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:49.611247 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad15f805-4c58-4931-b0fb-a34a2fb8e4ef-istio-envoy" pod "ad15f805-4c58-4931-b0fb-a34a2fb8e4ef" (UID: "ad15f805-4c58-4931-b0fb-a34a2fb8e4ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:42:49.707916 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:49.707877 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/136eabea-7a63-4d89-bcc1-0e3d6c8810cf-istio-data\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm\" (UID: \"136eabea-7a63-4d89-bcc1-0e3d6c8810cf\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm" Nov 02 01:42:49.708083 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:49.707957 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/136eabea-7a63-4d89-bcc1-0e3d6c8810cf-istiod-ca-cert\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm\" (UID: \"136eabea-7a63-4d89-bcc1-0e3d6c8810cf\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm" Nov 02 01:42:49.708083 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:49.708000 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/136eabea-7a63-4d89-bcc1-0e3d6c8810cf-credential-socket\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm\" (UID: \"136eabea-7a63-4d89-bcc1-0e3d6c8810cf\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm" Nov 02 01:42:49.708185 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:49.708137 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/136eabea-7a63-4d89-bcc1-0e3d6c8810cf-istio-podinfo\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm\" (UID: \"136eabea-7a63-4d89-bcc1-0e3d6c8810cf\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm" Nov 02 01:42:49.708231 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:49.708190 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/136eabea-7a63-4d89-bcc1-0e3d6c8810cf-istio-token\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm\" (UID: \"136eabea-7a63-4d89-bcc1-0e3d6c8810cf\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm" Nov 02 01:42:49.708282 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:49.708271 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/136eabea-7a63-4d89-bcc1-0e3d6c8810cf-workload-certs\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm\" (UID: \"136eabea-7a63-4d89-bcc1-0e3d6c8810cf\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm" Nov 02 01:42:49.708336 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:49.708307 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/136eabea-7a63-4d89-bcc1-0e3d6c8810cf-istio-data\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm\" (UID: \"136eabea-7a63-4d89-bcc1-0e3d6c8810cf\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm" Nov 02 01:42:49.708474 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:49.708449 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/136eabea-7a63-4d89-bcc1-0e3d6c8810cf-credential-socket\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm\" (UID: \"136eabea-7a63-4d89-bcc1-0e3d6c8810cf\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm" Nov 02 01:42:49.708559 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:49.708497 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/136eabea-7a63-4d89-bcc1-0e3d6c8810cf-workload-socket\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm\" (UID: \"136eabea-7a63-4d89-bcc1-0e3d6c8810cf\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm" Nov 02 01:42:49.708559 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:49.708550 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkhcn\" (UniqueName: \"kubernetes.io/projected/136eabea-7a63-4d89-bcc1-0e3d6c8810cf-kube-api-access-tkhcn\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm\" (UID: \"136eabea-7a63-4d89-bcc1-0e3d6c8810cf\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm" Nov 02 01:42:49.708670 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:49.708627 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/136eabea-7a63-4d89-bcc1-0e3d6c8810cf-workload-certs\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm\" (UID: \"136eabea-7a63-4d89-bcc1-0e3d6c8810cf\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm" Nov 02 01:42:49.708714 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:49.708682 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/136eabea-7a63-4d89-bcc1-0e3d6c8810cf-istio-envoy\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm\" (UID: \"136eabea-7a63-4d89-bcc1-0e3d6c8810cf\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm" Nov 02 01:42:49.708848 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:49.708815 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/136eabea-7a63-4d89-bcc1-0e3d6c8810cf-workload-socket\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm\" (UID: \"136eabea-7a63-4d89-bcc1-0e3d6c8810cf\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm" Nov 02 01:42:49.708848 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:49.708840 4427 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad15f805-4c58-4931-b0fb-a34a2fb8e4ef-istio-podinfo\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:42:49.708993 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:49.708827 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/136eabea-7a63-4d89-bcc1-0e3d6c8810cf-istiod-ca-cert\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm\" (UID: \"136eabea-7a63-4d89-bcc1-0e3d6c8810cf\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm" Nov 02 01:42:49.708993 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:49.708860 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w89ch\" (UniqueName: \"kubernetes.io/projected/ad15f805-4c58-4931-b0fb-a34a2fb8e4ef-kube-api-access-w89ch\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:42:49.709102 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:49.709028 4427 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad15f805-4c58-4931-b0fb-a34a2fb8e4ef-istio-envoy\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:42:49.709102 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:49.709082 4427 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad15f805-4c58-4931-b0fb-a34a2fb8e4ef-istio-token\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:42:49.710962 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:49.710935 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/136eabea-7a63-4d89-bcc1-0e3d6c8810cf-istio-podinfo\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm\" (UID: \"136eabea-7a63-4d89-bcc1-0e3d6c8810cf\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm" Nov 02 01:42:49.711070 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:49.710990 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/136eabea-7a63-4d89-bcc1-0e3d6c8810cf-istio-envoy\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm\" (UID: \"136eabea-7a63-4d89-bcc1-0e3d6c8810cf\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm" Nov 02 01:42:49.720547 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:49.720494 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/136eabea-7a63-4d89-bcc1-0e3d6c8810cf-istio-token\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm\" (UID: \"136eabea-7a63-4d89-bcc1-0e3d6c8810cf\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm" Nov 02 01:42:49.720816 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:49.720797 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkhcn\" (UniqueName: \"kubernetes.io/projected/136eabea-7a63-4d89-bcc1-0e3d6c8810cf-kube-api-access-tkhcn\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm\" (UID: \"136eabea-7a63-4d89-bcc1-0e3d6c8810cf\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm" Nov 02 01:42:49.813035 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:49.813004 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm" Nov 02 01:42:50.009059 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:50.008970 4427 patch_prober.go:29] interesting pod/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 01:42:50.009059 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:50.009049 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h" podUID="95975531-e9c3-4fcb-9f93-ddab47bb62ba" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 01:42:50.119273 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:42:50.119240 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod136eabea_7a63_4d89_bcc1_0e3d6c8810cf.slice/crio-ae0dac58e7a7f94ec3ad20c5e78f45c66b5c5d442ecbe8d670cdefdccd29d41f WatchSource:0}: Error finding container ae0dac58e7a7f94ec3ad20c5e78f45c66b5c5d442ecbe8d670cdefdccd29d41f: Status 404 returned error can't find the container with id ae0dac58e7a7f94ec3ad20c5e78f45c66b5c5d442ecbe8d670cdefdccd29d41f Nov 02 01:42:50.120640 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:50.120614 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm"] Nov 02 01:42:50.122556 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:50.122523 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:42:50.122657 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:50.122593 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:42:50.122657 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:50.122634 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:42:50.163841 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:50.163810 4427 generic.go:356] "Generic (PLEG): container finished" podID="ad15f805-4c58-4931-b0fb-a34a2fb8e4ef" containerID="534464f43bd3bb33393303f40960e784a343c9c11ff580c9c8b6ce68012f5c8d" exitCode=0 Nov 02 01:42:50.163988 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:50.163860 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km" Nov 02 01:42:50.163988 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:50.163901 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km" event={"ID":"ad15f805-4c58-4931-b0fb-a34a2fb8e4ef","Type":"ContainerDied","Data":"534464f43bd3bb33393303f40960e784a343c9c11ff580c9c8b6ce68012f5c8d"} Nov 02 01:42:50.163988 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:50.163966 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km" event={"ID":"ad15f805-4c58-4931-b0fb-a34a2fb8e4ef","Type":"ContainerDied","Data":"6ba5362ece87877d67c4539d5a5d526bd5e6c519f32f40fdeacf797d20ba3615"} Nov 02 01:42:50.163988 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:50.163989 4427 scope.go:122] "RemoveContainer" containerID="534464f43bd3bb33393303f40960e784a343c9c11ff580c9c8b6ce68012f5c8d" Nov 02 01:42:50.165608 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:50.165584 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm" event={"ID":"136eabea-7a63-4d89-bcc1-0e3d6c8810cf","Type":"ContainerStarted","Data":"ae0dac58e7a7f94ec3ad20c5e78f45c66b5c5d442ecbe8d670cdefdccd29d41f"} Nov 02 01:42:50.183579 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:50.182881 4427 scope.go:122] "RemoveContainer" containerID="534464f43bd3bb33393303f40960e784a343c9c11ff580c9c8b6ce68012f5c8d" Nov 02 01:42:50.183579 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:42:50.183257 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"534464f43bd3bb33393303f40960e784a343c9c11ff580c9c8b6ce68012f5c8d\": container with ID starting with 534464f43bd3bb33393303f40960e784a343c9c11ff580c9c8b6ce68012f5c8d not found: ID does not exist" containerID="534464f43bd3bb33393303f40960e784a343c9c11ff580c9c8b6ce68012f5c8d" Nov 02 01:42:50.183579 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:50.183284 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"534464f43bd3bb33393303f40960e784a343c9c11ff580c9c8b6ce68012f5c8d"} err="failed to get container status \"534464f43bd3bb33393303f40960e784a343c9c11ff580c9c8b6ce68012f5c8d\": rpc error: code = NotFound desc = could not find container \"534464f43bd3bb33393303f40960e784a343c9c11ff580c9c8b6ce68012f5c8d\": container with ID starting with 534464f43bd3bb33393303f40960e784a343c9c11ff580c9c8b6ce68012f5c8d not found: ID does not exist" Nov 02 01:42:50.234471 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:50.233275 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km"] Nov 02 01:42:50.272244 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:50.272212 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-edef28-openshift-default-5dc6c98847-w49km"] Nov 02 01:42:51.197612 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:51.197569 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm" event={"ID":"136eabea-7a63-4d89-bcc1-0e3d6c8810cf","Type":"ContainerStarted","Data":"e7bad34c068724ee09f318ec5119e242811f511e6d11437373e7b874c693f60a"} Nov 02 01:42:51.228565 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:51.228517 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm" podStartSLOduration=178.228504018 podStartE2EDuration="2m58.228504018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:39:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:42:51.221829063 +0000 UTC m=+1675.842834270" watchObservedRunningTime="2026-11-02 01:42:51.228504018 +0000 UTC m=+1675.849509223" Nov 02 01:42:51.540430 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:51.538914 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad15f805-4c58-4931-b0fb-a34a2fb8e4ef" path="/var/lib/kubelet/pods/ad15f805-4c58-4931-b0fb-a34a2fb8e4ef/volumes" Nov 02 01:42:51.813997 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:51.813880 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm" Nov 02 01:42:51.816125 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:51.815931 4427 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Nov 02 01:42:51.816125 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:51.815990 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm" podUID="136eabea-7a63-4d89-bcc1-0e3d6c8810cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Nov 02 01:42:51.946975 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:51.946947 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h" Nov 02 01:42:52.009904 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.007460 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95975531-e9c3-4fcb-9f93-ddab47bb62ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95975531-e9c3-4fcb-9f93-ddab47bb62ba-workload-certs\") pod \"95975531-e9c3-4fcb-9f93-ddab47bb62ba\" (UID: \"95975531-e9c3-4fcb-9f93-ddab47bb62ba\") " Nov 02 01:42:52.009904 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.007647 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95975531-e9c3-4fcb-9f93-ddab47bb62ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95975531-e9c3-4fcb-9f93-ddab47bb62ba-workload-socket\") pod \"95975531-e9c3-4fcb-9f93-ddab47bb62ba\" (UID: \"95975531-e9c3-4fcb-9f93-ddab47bb62ba\") " Nov 02 01:42:52.009904 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.007695 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95975531-e9c3-4fcb-9f93-ddab47bb62ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95975531-e9c3-4fcb-9f93-ddab47bb62ba-istio-envoy\") pod \"95975531-e9c3-4fcb-9f93-ddab47bb62ba\" (UID: \"95975531-e9c3-4fcb-9f93-ddab47bb62ba\") " Nov 02 01:42:52.009904 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.007742 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95975531-e9c3-4fcb-9f93-ddab47bb62ba-kube-api-access-qndzn\" (UniqueName: \"kubernetes.io/projected/95975531-e9c3-4fcb-9f93-ddab47bb62ba-kube-api-access-qndzn\") pod \"95975531-e9c3-4fcb-9f93-ddab47bb62ba\" (UID: \"95975531-e9c3-4fcb-9f93-ddab47bb62ba\") " Nov 02 01:42:52.009904 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.007749 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95975531-e9c3-4fcb-9f93-ddab47bb62ba-workload-certs" pod "95975531-e9c3-4fcb-9f93-ddab47bb62ba" (UID: "95975531-e9c3-4fcb-9f93-ddab47bb62ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:42:52.009904 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.007773 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95975531-e9c3-4fcb-9f93-ddab47bb62ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95975531-e9c3-4fcb-9f93-ddab47bb62ba-istiod-ca-cert\") pod \"95975531-e9c3-4fcb-9f93-ddab47bb62ba\" (UID: \"95975531-e9c3-4fcb-9f93-ddab47bb62ba\") " Nov 02 01:42:52.009904 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.007802 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95975531-e9c3-4fcb-9f93-ddab47bb62ba-istio-token\" (UniqueName: \"kubernetes.io/projected/95975531-e9c3-4fcb-9f93-ddab47bb62ba-istio-token\") pod \"95975531-e9c3-4fcb-9f93-ddab47bb62ba\" (UID: \"95975531-e9c3-4fcb-9f93-ddab47bb62ba\") " Nov 02 01:42:52.009904 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.007907 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/95975531-e9c3-4fcb-9f93-ddab47bb62ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95975531-e9c3-4fcb-9f93-ddab47bb62ba-istio-podinfo\") pod \"95975531-e9c3-4fcb-9f93-ddab47bb62ba\" (UID: \"95975531-e9c3-4fcb-9f93-ddab47bb62ba\") " Nov 02 01:42:52.009904 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.007973 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95975531-e9c3-4fcb-9f93-ddab47bb62ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95975531-e9c3-4fcb-9f93-ddab47bb62ba-istio-data\") pod \"95975531-e9c3-4fcb-9f93-ddab47bb62ba\" (UID: \"95975531-e9c3-4fcb-9f93-ddab47bb62ba\") " Nov 02 01:42:52.009904 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.008025 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95975531-e9c3-4fcb-9f93-ddab47bb62ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95975531-e9c3-4fcb-9f93-ddab47bb62ba-credential-socket\") pod \"95975531-e9c3-4fcb-9f93-ddab47bb62ba\" (UID: \"95975531-e9c3-4fcb-9f93-ddab47bb62ba\") " Nov 02 01:42:52.009904 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.009108 4427 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95975531-e9c3-4fcb-9f93-ddab47bb62ba-workload-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:42:52.009904 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.009326 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95975531-e9c3-4fcb-9f93-ddab47bb62ba-credential-socket" pod "95975531-e9c3-4fcb-9f93-ddab47bb62ba" (UID: "95975531-e9c3-4fcb-9f93-ddab47bb62ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:42:52.009904 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.009560 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95975531-e9c3-4fcb-9f93-ddab47bb62ba-istio-data" pod "95975531-e9c3-4fcb-9f93-ddab47bb62ba" (UID: "95975531-e9c3-4fcb-9f93-ddab47bb62ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:42:52.012740 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.012707 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95975531-e9c3-4fcb-9f93-ddab47bb62ba-istiod-ca-cert" pod "95975531-e9c3-4fcb-9f93-ddab47bb62ba" (UID: "95975531-e9c3-4fcb-9f93-ddab47bb62ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:42:52.013070 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.012971 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95975531-e9c3-4fcb-9f93-ddab47bb62ba-workload-socket" pod "95975531-e9c3-4fcb-9f93-ddab47bb62ba" (UID: "95975531-e9c3-4fcb-9f93-ddab47bb62ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:42:52.021668 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.015479 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj"] Nov 02 01:42:52.021668 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.016931 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/95975531-e9c3-4fcb-9f93-ddab47bb62ba-istio-podinfo" pod "95975531-e9c3-4fcb-9f93-ddab47bb62ba" (UID: "95975531-e9c3-4fcb-9f93-ddab47bb62ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:42:52.021668 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.021237 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95975531-e9c3-4fcb-9f93-ddab47bb62ba-kube-api-access-qndzn" pod "95975531-e9c3-4fcb-9f93-ddab47bb62ba" (UID: "95975531-e9c3-4fcb-9f93-ddab47bb62ba"). InnerVolumeSpecName "kube-api-access-qndzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:42:52.025070 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.022490 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95975531-e9c3-4fcb-9f93-ddab47bb62ba-istio-envoy" pod "95975531-e9c3-4fcb-9f93-ddab47bb62ba" (UID: "95975531-e9c3-4fcb-9f93-ddab47bb62ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:42:52.025070 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.022633 4427 cpu_manager.go:414] "containerMap: removing container" podUID="95975531-e9c3-4fcb-9f93-ddab47bb62ba" containerName="istio-proxy" Nov 02 01:42:52.025070 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.022656 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95975531-e9c3-4fcb-9f93-ddab47bb62ba" containerName="istio-proxy" Nov 02 01:42:52.025070 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.022909 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="95975531-e9c3-4fcb-9f93-ddab47bb62ba" containerName="istio-proxy" Nov 02 01:42:52.025070 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.022937 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="95975531-e9c3-4fcb-9f93-ddab47bb62ba" containerName="istio-proxy" Nov 02 01:42:52.029493 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.028533 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj" Nov 02 01:42:52.032950 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.032918 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95975531-e9c3-4fcb-9f93-ddab47bb62ba-istio-token" pod "95975531-e9c3-4fcb-9f93-ddab47bb62ba" (UID: "95975531-e9c3-4fcb-9f93-ddab47bb62ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:42:52.035952 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.035931 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj"] Nov 02 01:42:52.110564 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.110529 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36cac416-8352-4653-8ec9-7ad64348afa7-credential-socket\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj\" (UID: \"36cac416-8352-4653-8ec9-7ad64348afa7\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj" Nov 02 01:42:52.110736 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.110571 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-725hj\" (UniqueName: \"kubernetes.io/projected/36cac416-8352-4653-8ec9-7ad64348afa7-kube-api-access-725hj\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj\" (UID: \"36cac416-8352-4653-8ec9-7ad64348afa7\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj" Nov 02 01:42:52.110736 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.110598 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36cac416-8352-4653-8ec9-7ad64348afa7-workload-certs\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj\" (UID: \"36cac416-8352-4653-8ec9-7ad64348afa7\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj" Nov 02 01:42:52.110736 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.110689 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36cac416-8352-4653-8ec9-7ad64348afa7-istio-token\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj\" (UID: \"36cac416-8352-4653-8ec9-7ad64348afa7\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj" Nov 02 01:42:52.110877 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.110827 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36cac416-8352-4653-8ec9-7ad64348afa7-istio-podinfo\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj\" (UID: \"36cac416-8352-4653-8ec9-7ad64348afa7\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj" Nov 02 01:42:52.110984 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.110967 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36cac416-8352-4653-8ec9-7ad64348afa7-workload-socket\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj\" (UID: \"36cac416-8352-4653-8ec9-7ad64348afa7\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj" Nov 02 01:42:52.111090 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.111077 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36cac416-8352-4653-8ec9-7ad64348afa7-istio-envoy\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj\" (UID: \"36cac416-8352-4653-8ec9-7ad64348afa7\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj" Nov 02 01:42:52.111164 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.111118 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36cac416-8352-4653-8ec9-7ad64348afa7-istio-data\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj\" (UID: \"36cac416-8352-4653-8ec9-7ad64348afa7\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj" Nov 02 01:42:52.111268 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.111251 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36cac416-8352-4653-8ec9-7ad64348afa7-istiod-ca-cert\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj\" (UID: \"36cac416-8352-4653-8ec9-7ad64348afa7\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj" Nov 02 01:42:52.111459 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.111444 4427 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95975531-e9c3-4fcb-9f93-ddab47bb62ba-istio-envoy\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:42:52.111568 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.111461 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qndzn\" (UniqueName: \"kubernetes.io/projected/95975531-e9c3-4fcb-9f93-ddab47bb62ba-kube-api-access-qndzn\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:42:52.111568 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.111477 4427 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95975531-e9c3-4fcb-9f93-ddab47bb62ba-istiod-ca-cert\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:42:52.111568 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.111487 4427 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95975531-e9c3-4fcb-9f93-ddab47bb62ba-istio-token\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:42:52.111568 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.111497 4427 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95975531-e9c3-4fcb-9f93-ddab47bb62ba-istio-podinfo\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:42:52.111568 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.111505 4427 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95975531-e9c3-4fcb-9f93-ddab47bb62ba-istio-data\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:42:52.111568 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.111515 4427 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95975531-e9c3-4fcb-9f93-ddab47bb62ba-credential-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:42:52.111568 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.111525 4427 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95975531-e9c3-4fcb-9f93-ddab47bb62ba-workload-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:42:52.210707 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.210669 4427 generic.go:356] "Generic (PLEG): container finished" podID="95975531-e9c3-4fcb-9f93-ddab47bb62ba" containerID="67f8e1110ac77d3bcdbc8748e701a2e277e503141cc9103464ad79bec919c73c" exitCode=0 Nov 02 01:42:52.211202 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.210739 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h" Nov 02 01:42:52.211202 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.210776 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h" event={"ID":"95975531-e9c3-4fcb-9f93-ddab47bb62ba","Type":"ContainerDied","Data":"67f8e1110ac77d3bcdbc8748e701a2e277e503141cc9103464ad79bec919c73c"} Nov 02 01:42:52.211202 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.210834 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h" event={"ID":"95975531-e9c3-4fcb-9f93-ddab47bb62ba","Type":"ContainerDied","Data":"5aae6a4df1b75decabd9210387438af2add7061575bda9af6e20125c069d5b9e"} Nov 02 01:42:52.211202 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.210859 4427 scope.go:122] "RemoveContainer" containerID="67f8e1110ac77d3bcdbc8748e701a2e277e503141cc9103464ad79bec919c73c" Nov 02 01:42:52.212892 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.212869 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36cac416-8352-4653-8ec9-7ad64348afa7-istio-envoy\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj\" (UID: \"36cac416-8352-4653-8ec9-7ad64348afa7\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj" Nov 02 01:42:52.213063 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.212917 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36cac416-8352-4653-8ec9-7ad64348afa7-istio-data\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj\" (UID: \"36cac416-8352-4653-8ec9-7ad64348afa7\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj" Nov 02 01:42:52.213063 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.212985 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36cac416-8352-4653-8ec9-7ad64348afa7-istiod-ca-cert\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj\" (UID: \"36cac416-8352-4653-8ec9-7ad64348afa7\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj" Nov 02 01:42:52.213063 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.213059 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36cac416-8352-4653-8ec9-7ad64348afa7-credential-socket\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj\" (UID: \"36cac416-8352-4653-8ec9-7ad64348afa7\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj" Nov 02 01:42:52.213241 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.213162 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-725hj\" (UniqueName: \"kubernetes.io/projected/36cac416-8352-4653-8ec9-7ad64348afa7-kube-api-access-725hj\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj\" (UID: \"36cac416-8352-4653-8ec9-7ad64348afa7\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj" Nov 02 01:42:52.213241 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.213234 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36cac416-8352-4653-8ec9-7ad64348afa7-workload-certs\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj\" (UID: \"36cac416-8352-4653-8ec9-7ad64348afa7\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj" Nov 02 01:42:52.213349 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.213282 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36cac416-8352-4653-8ec9-7ad64348afa7-istio-token\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj\" (UID: \"36cac416-8352-4653-8ec9-7ad64348afa7\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj" Nov 02 01:42:52.213499 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.213444 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36cac416-8352-4653-8ec9-7ad64348afa7-istio-data\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj\" (UID: \"36cac416-8352-4653-8ec9-7ad64348afa7\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj" Nov 02 01:42:52.213499 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.213456 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36cac416-8352-4653-8ec9-7ad64348afa7-istio-podinfo\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj\" (UID: \"36cac416-8352-4653-8ec9-7ad64348afa7\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj" Nov 02 01:42:52.213616 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.213553 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36cac416-8352-4653-8ec9-7ad64348afa7-workload-socket\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj\" (UID: \"36cac416-8352-4653-8ec9-7ad64348afa7\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj" Nov 02 01:42:52.213828 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.213806 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36cac416-8352-4653-8ec9-7ad64348afa7-istiod-ca-cert\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj\" (UID: \"36cac416-8352-4653-8ec9-7ad64348afa7\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj" Nov 02 01:42:52.213940 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.213832 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36cac416-8352-4653-8ec9-7ad64348afa7-credential-socket\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj\" (UID: \"36cac416-8352-4653-8ec9-7ad64348afa7\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj" Nov 02 01:42:52.213994 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.213939 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36cac416-8352-4653-8ec9-7ad64348afa7-workload-socket\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj\" (UID: \"36cac416-8352-4653-8ec9-7ad64348afa7\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj" Nov 02 01:42:52.214039 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.214012 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36cac416-8352-4653-8ec9-7ad64348afa7-workload-certs\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj\" (UID: \"36cac416-8352-4653-8ec9-7ad64348afa7\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj" Nov 02 01:42:52.216061 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.216042 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36cac416-8352-4653-8ec9-7ad64348afa7-istio-envoy\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj\" (UID: \"36cac416-8352-4653-8ec9-7ad64348afa7\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj" Nov 02 01:42:52.216561 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.216538 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36cac416-8352-4653-8ec9-7ad64348afa7-istio-podinfo\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj\" (UID: \"36cac416-8352-4653-8ec9-7ad64348afa7\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj" Nov 02 01:42:52.225426 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.225242 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36cac416-8352-4653-8ec9-7ad64348afa7-istio-token\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj\" (UID: \"36cac416-8352-4653-8ec9-7ad64348afa7\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj" Nov 02 01:42:52.225426 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.225396 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-725hj\" (UniqueName: \"kubernetes.io/projected/36cac416-8352-4653-8ec9-7ad64348afa7-kube-api-access-725hj\") pod \"e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj\" (UID: \"36cac416-8352-4653-8ec9-7ad64348afa7\") " pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj" Nov 02 01:42:52.225596 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.225502 4427 scope.go:122] "RemoveContainer" containerID="e75bccb167e5e1f6882788fe66536261d715bbce3d95a0d95191a10c531b4863" Nov 02 01:42:52.237812 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.237772 4427 scope.go:122] "RemoveContainer" containerID="67f8e1110ac77d3bcdbc8748e701a2e277e503141cc9103464ad79bec919c73c" Nov 02 01:42:52.239106 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:42:52.239072 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67f8e1110ac77d3bcdbc8748e701a2e277e503141cc9103464ad79bec919c73c\": container with ID starting with 67f8e1110ac77d3bcdbc8748e701a2e277e503141cc9103464ad79bec919c73c not found: ID does not exist" containerID="67f8e1110ac77d3bcdbc8748e701a2e277e503141cc9103464ad79bec919c73c" Nov 02 01:42:52.239234 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.239118 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67f8e1110ac77d3bcdbc8748e701a2e277e503141cc9103464ad79bec919c73c"} err="failed to get container status \"67f8e1110ac77d3bcdbc8748e701a2e277e503141cc9103464ad79bec919c73c\": rpc error: code = NotFound desc = could not find container \"67f8e1110ac77d3bcdbc8748e701a2e277e503141cc9103464ad79bec919c73c\": container with ID starting with 67f8e1110ac77d3bcdbc8748e701a2e277e503141cc9103464ad79bec919c73c not found: ID does not exist" Nov 02 01:42:52.239234 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.239144 4427 scope.go:122] "RemoveContainer" containerID="e75bccb167e5e1f6882788fe66536261d715bbce3d95a0d95191a10c531b4863" Nov 02 01:42:52.240813 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:42:52.240782 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e75bccb167e5e1f6882788fe66536261d715bbce3d95a0d95191a10c531b4863\": container with ID starting with e75bccb167e5e1f6882788fe66536261d715bbce3d95a0d95191a10c531b4863 not found: ID does not exist" containerID="e75bccb167e5e1f6882788fe66536261d715bbce3d95a0d95191a10c531b4863" Nov 02 01:42:52.240989 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.240851 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e75bccb167e5e1f6882788fe66536261d715bbce3d95a0d95191a10c531b4863"} err="failed to get container status \"e75bccb167e5e1f6882788fe66536261d715bbce3d95a0d95191a10c531b4863\": rpc error: code = NotFound desc = could not find container \"e75bccb167e5e1f6882788fe66536261d715bbce3d95a0d95191a10c531b4863\": container with ID starting with e75bccb167e5e1f6882788fe66536261d715bbce3d95a0d95191a10c531b4863 not found: ID does not exist" Nov 02 01:42:52.249581 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.249558 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h"] Nov 02 01:42:52.254592 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.254571 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-8b3bae-openshift-default-79f7d5c45c-mbq9h"] Nov 02 01:42:52.404404 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.404199 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj" Nov 02 01:42:52.738159 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.738128 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj"] Nov 02 01:42:52.738431 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:42:52.738358 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36cac416_8352_4653_8ec9_7ad64348afa7.slice/crio-9e0e090a19e1d65affa6fab6ccfd8a8f492bf848b1476ec9c4ca525c4b1dc602 WatchSource:0}: Error finding container 9e0e090a19e1d65affa6fab6ccfd8a8f492bf848b1476ec9c4ca525c4b1dc602: Status 404 returned error can't find the container with id 9e0e090a19e1d65affa6fab6ccfd8a8f492bf848b1476ec9c4ca525c4b1dc602 Nov 02 01:42:52.741860 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.741822 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:42:52.741968 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.741949 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:42:52.742016 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.742002 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:42:52.817399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.813996 4427 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Nov 02 01:42:52.817399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:52.814089 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm" podUID="136eabea-7a63-4d89-bcc1-0e3d6c8810cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Nov 02 01:42:53.223132 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:53.223088 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj" event={"ID":"36cac416-8352-4653-8ec9-7ad64348afa7","Type":"ContainerStarted","Data":"cf5643a4140429f936a509f8ea84de25169cef9edca00cf73746c2659bd2a183"} Nov 02 01:42:53.223593 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:53.223154 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj" event={"ID":"36cac416-8352-4653-8ec9-7ad64348afa7","Type":"ContainerStarted","Data":"9e0e090a19e1d65affa6fab6ccfd8a8f492bf848b1476ec9c4ca525c4b1dc602"} Nov 02 01:42:53.297021 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:53.296947 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj" podStartSLOduration=165.296919907 podStartE2EDuration="2m45.296919907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:40:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:42:53.249825221 +0000 UTC m=+1677.870830445" watchObservedRunningTime="2026-11-02 01:42:53.296919907 +0000 UTC m=+1677.917925119" Nov 02 01:42:53.405925 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:53.405886 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj" Nov 02 01:42:53.408095 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:53.408068 4427 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 02 01:42:53.408236 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:53.408123 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj" podUID="36cac416-8352-4653-8ec9-7ad64348afa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 02 01:42:53.553554 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:53.553443 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95975531-e9c3-4fcb-9f93-ddab47bb62ba" path="/var/lib/kubelet/pods/95975531-e9c3-4fcb-9f93-ddab47bb62ba/volumes" Nov 02 01:42:53.816972 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:53.816865 4427 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Nov 02 01:42:53.816972 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:53.816930 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm" podUID="136eabea-7a63-4d89-bcc1-0e3d6c8810cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Nov 02 01:42:54.407401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:54.406744 4427 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 02 01:42:54.407401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:54.406808 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj" podUID="36cac416-8352-4653-8ec9-7ad64348afa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 02 01:42:54.816504 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:54.816355 4427 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Nov 02 01:42:54.816690 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:54.816497 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm" podUID="136eabea-7a63-4d89-bcc1-0e3d6c8810cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Nov 02 01:42:55.413457 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:55.412269 4427 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 02 01:42:55.413457 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:55.412348 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj" podUID="36cac416-8352-4653-8ec9-7ad64348afa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 02 01:42:55.814252 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:55.813699 4427 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Nov 02 01:42:55.814252 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:55.813765 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm" podUID="136eabea-7a63-4d89-bcc1-0e3d6c8810cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Nov 02 01:42:56.406904 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:56.406866 4427 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 02 01:42:56.407186 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:56.407158 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj" podUID="36cac416-8352-4653-8ec9-7ad64348afa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 02 01:42:56.940418 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:56.894587 4427 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Nov 02 01:42:56.940418 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:56.894652 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm" podUID="136eabea-7a63-4d89-bcc1-0e3d6c8810cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Nov 02 01:42:57.474284 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:57.474252 4427 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 02 01:42:57.474686 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:57.474656 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj" podUID="36cac416-8352-4653-8ec9-7ad64348afa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 02 01:42:58.185932 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:58.185896 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm" Nov 02 01:42:58.192613 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:58.192586 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm" Nov 02 01:42:58.194614 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:58.194590 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm" Nov 02 01:42:58.404997 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:58.404958 4427 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 02 01:42:58.405170 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:58.405031 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj" podUID="36cac416-8352-4653-8ec9-7ad64348afa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 02 01:42:59.410598 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:59.410565 4427 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 02 01:42:59.411209 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:42:59.411174 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj" podUID="36cac416-8352-4653-8ec9-7ad64348afa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 02 01:43:00.374752 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:00.374706 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-0bbeda/test-model-0bbeda-kserve-5cb56fbff9-lqp9j" podUID="b45e955d-0d37-44c2-96be-42dc39687913" containerName="main" containerID="cri-o://e34deec8bd31da8db32c1f3335f3a295677a0550b02aa961c660116bf80b680f" gracePeriod=44 Nov 02 01:43:00.511169 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:00.511002 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj" Nov 02 01:43:00.512602 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:00.512586 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj" Nov 02 01:43:00.513530 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:00.513479 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj" Nov 02 01:43:00.634293 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:00.633435 4427 generic.go:356] "Generic (PLEG): container finished" podID="b45e955d-0d37-44c2-96be-42dc39687913" containerID="e34deec8bd31da8db32c1f3335f3a295677a0550b02aa961c660116bf80b680f" exitCode=0 Nov 02 01:43:00.635643 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:00.634506 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0bbeda/test-model-0bbeda-kserve-5cb56fbff9-lqp9j" event={"ID":"b45e955d-0d37-44c2-96be-42dc39687913","Type":"ContainerDied","Data":"e34deec8bd31da8db32c1f3335f3a295677a0550b02aa961c660116bf80b680f"} Nov 02 01:43:00.863229 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:00.863203 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0bbeda/test-model-0bbeda-kserve-5cb56fbff9-lqp9j" Nov 02 01:43:01.046994 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:01.046907 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b45e955d-0d37-44c2-96be-42dc39687913-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b45e955d-0d37-44c2-96be-42dc39687913-tmp-dir\") pod \"b45e955d-0d37-44c2-96be-42dc39687913\" (UID: \"b45e955d-0d37-44c2-96be-42dc39687913\") " Nov 02 01:43:01.047196 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:01.046997 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b45e955d-0d37-44c2-96be-42dc39687913-tls-certs\" (UniqueName: \"kubernetes.io/secret/b45e955d-0d37-44c2-96be-42dc39687913-tls-certs\") pod \"b45e955d-0d37-44c2-96be-42dc39687913\" (UID: \"b45e955d-0d37-44c2-96be-42dc39687913\") " Nov 02 01:43:01.047196 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:01.047102 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b45e955d-0d37-44c2-96be-42dc39687913-home\" (UniqueName: \"kubernetes.io/empty-dir/b45e955d-0d37-44c2-96be-42dc39687913-home\") pod \"b45e955d-0d37-44c2-96be-42dc39687913\" (UID: \"b45e955d-0d37-44c2-96be-42dc39687913\") " Nov 02 01:43:01.047196 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:01.047136 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b45e955d-0d37-44c2-96be-42dc39687913-kube-api-access-k6bpc\" (UniqueName: \"kubernetes.io/projected/b45e955d-0d37-44c2-96be-42dc39687913-kube-api-access-k6bpc\") pod \"b45e955d-0d37-44c2-96be-42dc39687913\" (UID: \"b45e955d-0d37-44c2-96be-42dc39687913\") " Nov 02 01:43:01.047196 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:01.047163 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b45e955d-0d37-44c2-96be-42dc39687913-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b45e955d-0d37-44c2-96be-42dc39687913-dshm\") pod \"b45e955d-0d37-44c2-96be-42dc39687913\" (UID: \"b45e955d-0d37-44c2-96be-42dc39687913\") " Nov 02 01:43:01.047471 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:01.047191 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b45e955d-0d37-44c2-96be-42dc39687913-tmp-dir" pod "b45e955d-0d37-44c2-96be-42dc39687913" (UID: "b45e955d-0d37-44c2-96be-42dc39687913"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:01.047471 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:01.047345 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b45e955d-0d37-44c2-96be-42dc39687913-home" pod "b45e955d-0d37-44c2-96be-42dc39687913" (UID: "b45e955d-0d37-44c2-96be-42dc39687913"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:01.047555 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:01.047534 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b45e955d-0d37-44c2-96be-42dc39687913-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b45e955d-0d37-44c2-96be-42dc39687913-model-cache\") pod \"b45e955d-0d37-44c2-96be-42dc39687913\" (UID: \"b45e955d-0d37-44c2-96be-42dc39687913\") " Nov 02 01:43:01.048018 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:01.047896 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b45e955d-0d37-44c2-96be-42dc39687913-model-cache" pod "b45e955d-0d37-44c2-96be-42dc39687913" (UID: "b45e955d-0d37-44c2-96be-42dc39687913"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:01.048855 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:01.048831 4427 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b45e955d-0d37-44c2-96be-42dc39687913-model-cache\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:01.048960 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:01.048860 4427 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b45e955d-0d37-44c2-96be-42dc39687913-tmp-dir\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:01.048960 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:01.048876 4427 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b45e955d-0d37-44c2-96be-42dc39687913-home\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:01.049863 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:01.049842 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b45e955d-0d37-44c2-96be-42dc39687913-tls-certs" pod "b45e955d-0d37-44c2-96be-42dc39687913" (UID: "b45e955d-0d37-44c2-96be-42dc39687913"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:43:01.050493 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:01.050471 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b45e955d-0d37-44c2-96be-42dc39687913-dshm" pod "b45e955d-0d37-44c2-96be-42dc39687913" (UID: "b45e955d-0d37-44c2-96be-42dc39687913"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:01.050602 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:01.050581 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b45e955d-0d37-44c2-96be-42dc39687913-kube-api-access-k6bpc" pod "b45e955d-0d37-44c2-96be-42dc39687913" (UID: "b45e955d-0d37-44c2-96be-42dc39687913"). InnerVolumeSpecName "kube-api-access-k6bpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:43:01.150830 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:01.150797 4427 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b45e955d-0d37-44c2-96be-42dc39687913-tls-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:01.150830 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:01.150825 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6bpc\" (UniqueName: \"kubernetes.io/projected/b45e955d-0d37-44c2-96be-42dc39687913-kube-api-access-k6bpc\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:01.150830 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:01.150835 4427 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b45e955d-0d37-44c2-96be-42dc39687913-dshm\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:01.607013 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:43:01.606896 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:43:01.649747 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:01.649712 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0bbeda/test-model-0bbeda-kserve-5cb56fbff9-lqp9j" Nov 02 01:43:01.649943 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:01.649717 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0bbeda/test-model-0bbeda-kserve-5cb56fbff9-lqp9j" event={"ID":"b45e955d-0d37-44c2-96be-42dc39687913","Type":"ContainerDied","Data":"c3f4fc28cdb2e7290c6e628761636139def59a99543bead180a8fd7945d1e3cc"} Nov 02 01:43:01.649943 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:01.649854 4427 scope.go:122] "RemoveContainer" containerID="e34deec8bd31da8db32c1f3335f3a295677a0550b02aa961c660116bf80b680f" Nov 02 01:43:01.675935 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:01.675879 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-0bbeda/test-model-0bbeda-kserve-5cb56fbff9-lqp9j"] Nov 02 01:43:01.684210 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:01.684181 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-0bbeda/test-model-0bbeda-kserve-5cb56fbff9-lqp9j"] Nov 02 01:43:01.757410 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:01.754606 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-0bbeda/test-model-0bbeda-kserve-5cb56fbff9-lqp9j" podUID="b45e955d-0d37-44c2-96be-42dc39687913" containerName="main" probeResult="failure" output="Get \"https://10.133.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:43:03.537820 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:03.537783 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b45e955d-0d37-44c2-96be-42dc39687913" path="/var/lib/kubelet/pods/b45e955d-0d37-44c2-96be-42dc39687913/volumes" Nov 02 01:43:09.655897 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:09.655857 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-addad9-7579959b99-l926m"] Nov 02 01:43:14.656439 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:14.656361 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-addad9-7579959b99-l926m" podUID="3d9b8d38-b3ce-4556-8580-93cfd36b2d2a" containerName="payload-pre-processing" containerID="cri-o://ef8e82b2a4a793ac0975e87e30e9b105c4d4e8ed64ae72f586943d4999e84141" gracePeriod=25 Nov 02 01:43:17.756298 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:17.756261 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e412d5-68746dfdfc-lrhnf"] Nov 02 01:43:17.772517 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:17.772484 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e412d5-5ff4cd49ff-8fb8p"] Nov 02 01:43:21.198336 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:21.198296 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-addad9-7579959b99-l926m" podUID="3d9b8d38-b3ce-4556-8580-93cfd36b2d2a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 01:43:22.215554 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:22.215512 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b34fc0-6cd9f9c5fc-9n8zq"] Nov 02 01:43:22.231115 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:22.231078 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b34fc0-5b786c7bdc-8lrhc"] Nov 02 01:43:22.778537 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:22.778459 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e412d5-5ff4cd49ff-8fb8p" podUID="5ff70346-02a9-4a43-9b5f-19a83d64d169" containerName="payload-pre-processing" containerID="cri-o://a35b3934809728a190648c59983a82ef5be7457184fa58a425b5c1a013284bcb" gracePeriod=25 Nov 02 01:43:23.247730 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:23.247701 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e412d5-5ff4cd49ff-8fb8p" Nov 02 01:43:23.388401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:23.387346 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-e412d5-68746dfdfc-lrhnf" podUID="8f3cd62e-2f33-481d-a827-63d2fddf6a46" containerName="payload-processing" containerID="cri-o://1ef6ed1b4872fbbad3e46a725197598d9776cff39bc991286854717a2b8ccc7a" gracePeriod=25 Nov 02 01:43:23.394211 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:23.393934 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ff70346-02a9-4a43-9b5f-19a83d64d169-kube-api-access-2j96j\" (UniqueName: \"kubernetes.io/projected/5ff70346-02a9-4a43-9b5f-19a83d64d169-kube-api-access-2j96j\") pod \"5ff70346-02a9-4a43-9b5f-19a83d64d169\" (UID: \"5ff70346-02a9-4a43-9b5f-19a83d64d169\") " Nov 02 01:43:23.395199 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:23.394261 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ff70346-02a9-4a43-9b5f-19a83d64d169-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ff70346-02a9-4a43-9b5f-19a83d64d169-plugins-config-volume\") pod \"5ff70346-02a9-4a43-9b5f-19a83d64d169\" (UID: \"5ff70346-02a9-4a43-9b5f-19a83d64d169\") " Nov 02 01:43:23.395777 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:23.395747 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ff70346-02a9-4a43-9b5f-19a83d64d169-plugins-config-volume" pod "5ff70346-02a9-4a43-9b5f-19a83d64d169" (UID: "5ff70346-02a9-4a43-9b5f-19a83d64d169"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:43:23.411170 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:23.411133 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ff70346-02a9-4a43-9b5f-19a83d64d169-kube-api-access-2j96j" pod "5ff70346-02a9-4a43-9b5f-19a83d64d169" (UID: "5ff70346-02a9-4a43-9b5f-19a83d64d169"). InnerVolumeSpecName "kube-api-access-2j96j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:43:23.497176 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:23.497134 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2j96j\" (UniqueName: \"kubernetes.io/projected/5ff70346-02a9-4a43-9b5f-19a83d64d169-kube-api-access-2j96j\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:23.497176 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:23.497168 4427 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ff70346-02a9-4a43-9b5f-19a83d64d169-plugins-config-volume\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:23.796973 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:23.796947 4427 generic.go:356] "Generic (PLEG): container finished" podID="8f3cd62e-2f33-481d-a827-63d2fddf6a46" containerID="1ef6ed1b4872fbbad3e46a725197598d9776cff39bc991286854717a2b8ccc7a" exitCode=0 Nov 02 01:43:23.797240 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:23.797223 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e412d5-68746dfdfc-lrhnf" event={"ID":"8f3cd62e-2f33-481d-a827-63d2fddf6a46","Type":"ContainerDied","Data":"1ef6ed1b4872fbbad3e46a725197598d9776cff39bc991286854717a2b8ccc7a"} Nov 02 01:43:23.800011 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:23.799982 4427 generic.go:356] "Generic (PLEG): container finished" podID="5ff70346-02a9-4a43-9b5f-19a83d64d169" containerID="a35b3934809728a190648c59983a82ef5be7457184fa58a425b5c1a013284bcb" exitCode=0 Nov 02 01:43:23.800351 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:23.800326 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e412d5-5ff4cd49ff-8fb8p" event={"ID":"5ff70346-02a9-4a43-9b5f-19a83d64d169","Type":"ContainerDied","Data":"a35b3934809728a190648c59983a82ef5be7457184fa58a425b5c1a013284bcb"} Nov 02 01:43:23.800447 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:23.800412 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e412d5-5ff4cd49ff-8fb8p" event={"ID":"5ff70346-02a9-4a43-9b5f-19a83d64d169","Type":"ContainerDied","Data":"f439ce7b1b32f47081a1395a6d21355b26b8ad4c5992bd90d2ec9b05ff6e006e"} Nov 02 01:43:23.800447 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:23.800443 4427 scope.go:122] "RemoveContainer" containerID="a35b3934809728a190648c59983a82ef5be7457184fa58a425b5c1a013284bcb" Nov 02 01:43:23.800661 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:23.800647 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e412d5-5ff4cd49ff-8fb8p" Nov 02 01:43:23.817951 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:23.817932 4427 scope.go:122] "RemoveContainer" containerID="a35b3934809728a190648c59983a82ef5be7457184fa58a425b5c1a013284bcb" Nov 02 01:43:23.818194 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:43:23.818174 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a35b3934809728a190648c59983a82ef5be7457184fa58a425b5c1a013284bcb\": container with ID starting with a35b3934809728a190648c59983a82ef5be7457184fa58a425b5c1a013284bcb not found: ID does not exist" containerID="a35b3934809728a190648c59983a82ef5be7457184fa58a425b5c1a013284bcb" Nov 02 01:43:23.818264 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:23.818202 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a35b3934809728a190648c59983a82ef5be7457184fa58a425b5c1a013284bcb"} err="failed to get container status \"a35b3934809728a190648c59983a82ef5be7457184fa58a425b5c1a013284bcb\": rpc error: code = NotFound desc = could not find container \"a35b3934809728a190648c59983a82ef5be7457184fa58a425b5c1a013284bcb\": container with ID starting with a35b3934809728a190648c59983a82ef5be7457184fa58a425b5c1a013284bcb not found: ID does not exist" Nov 02 01:43:23.823632 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:23.823599 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e412d5-68746dfdfc-lrhnf" Nov 02 01:43:23.845474 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:23.845443 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e412d5-5ff4cd49ff-8fb8p"] Nov 02 01:43:23.851515 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:23.851488 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e412d5-5ff4cd49ff-8fb8p"] Nov 02 01:43:23.949972 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:23.949933 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f3cd62e-2f33-481d-a827-63d2fddf6a46-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f3cd62e-2f33-481d-a827-63d2fddf6a46-plugins-config-volume\") pod \"8f3cd62e-2f33-481d-a827-63d2fddf6a46\" (UID: \"8f3cd62e-2f33-481d-a827-63d2fddf6a46\") " Nov 02 01:43:23.950276 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:23.950253 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f3cd62e-2f33-481d-a827-63d2fddf6a46-kube-api-access-9s6kj\" (UniqueName: \"kubernetes.io/projected/8f3cd62e-2f33-481d-a827-63d2fddf6a46-kube-api-access-9s6kj\") pod \"8f3cd62e-2f33-481d-a827-63d2fddf6a46\" (UID: \"8f3cd62e-2f33-481d-a827-63d2fddf6a46\") " Nov 02 01:43:23.950390 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:23.950351 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f3cd62e-2f33-481d-a827-63d2fddf6a46-plugins-config-volume" pod "8f3cd62e-2f33-481d-a827-63d2fddf6a46" (UID: "8f3cd62e-2f33-481d-a827-63d2fddf6a46"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:43:23.951167 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:23.951143 4427 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f3cd62e-2f33-481d-a827-63d2fddf6a46-plugins-config-volume\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:23.952864 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:23.952836 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f3cd62e-2f33-481d-a827-63d2fddf6a46-kube-api-access-9s6kj" pod "8f3cd62e-2f33-481d-a827-63d2fddf6a46" (UID: "8f3cd62e-2f33-481d-a827-63d2fddf6a46"). InnerVolumeSpecName "kube-api-access-9s6kj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:43:24.052807 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:24.052722 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s6kj\" (UniqueName: \"kubernetes.io/projected/8f3cd62e-2f33-481d-a827-63d2fddf6a46-kube-api-access-9s6kj\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:24.545617 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:24.545216 4427 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w" secret="" err="secret \"e2e-inf-b-14b63f-openshift-default-dockercfg-kkc76\" not found" Nov 02 01:43:24.548405 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:24.548247 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w"] Nov 02 01:43:25.054730 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:25.054684 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e412d5-68746dfdfc-lrhnf" event={"ID":"8f3cd62e-2f33-481d-a827-63d2fddf6a46","Type":"ContainerDied","Data":"556a34a1708cbb534de86a5e6f2a71ca6a6656575e73e9ed66bef56b970871e7"} Nov 02 01:43:25.055074 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:25.055055 4427 scope.go:122] "RemoveContainer" containerID="1ef6ed1b4872fbbad3e46a725197598d9776cff39bc991286854717a2b8ccc7a" Nov 02 01:43:25.056264 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:25.056245 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e412d5-68746dfdfc-lrhnf" Nov 02 01:43:25.057692 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:25.057620 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w" podUID="51776e18-2d31-454f-8c33-d7fbb169c601" containerName="istio-proxy" containerID="cri-o://d54bd0fa45fe5c2c3e233a10fa99a4c2028bc5b352bf4cb393d2d486a7783358" gracePeriod=30 Nov 02 01:43:25.107215 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:25.107178 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e412d5-68746dfdfc-lrhnf"] Nov 02 01:43:25.112597 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:25.112573 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e412d5-68746dfdfc-lrhnf"] Nov 02 01:43:25.539931 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:25.539897 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ff70346-02a9-4a43-9b5f-19a83d64d169" path="/var/lib/kubelet/pods/5ff70346-02a9-4a43-9b5f-19a83d64d169/volumes" Nov 02 01:43:25.540582 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:25.540561 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f3cd62e-2f33-481d-a827-63d2fddf6a46" path="/var/lib/kubelet/pods/8f3cd62e-2f33-481d-a827-63d2fddf6a46/volumes" Nov 02 01:43:26.264747 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:26.264711 4427 patch_prober.go:29] interesting pod/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 01:43:26.265437 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:26.265401 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w" podUID="51776e18-2d31-454f-8c33-d7fbb169c601" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 01:43:26.515754 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:26.515639 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8"] Nov 02 01:43:26.516220 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:26.516164 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8" podUID="a6fe33af-426d-4155-98a1-a6dfdaeead32" containerName="istio-proxy" containerID="cri-o://a0db62a8741afd6232db998ff821a91006a2990a18326b967f8a037f4bd7e8d4" gracePeriod=30 Nov 02 01:43:27.216338 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:27.216276 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-b34fc0-6cd9f9c5fc-9n8zq" podUID="1e58e874-db28-4a0e-8b46-368b6d4d0960" containerName="payload-processing" containerID="cri-o://d65fa69dbf28c35abecfbb9bcfc5765dc93643496377c7228266dd0edfc22ec0" gracePeriod=25 Nov 02 01:43:27.232549 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:27.232502 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b34fc0-5b786c7bdc-8lrhc" podUID="d10ccd31-3d14-498e-bf7a-0b3034576528" containerName="payload-pre-processing" containerID="cri-o://ba8d5bfd013544907336604267b6db353fb410700c119045e616171898fb616a" gracePeriod=25 Nov 02 01:43:29.162399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:29.152576 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-b34fc0-6cd9f9c5fc-9n8zq" podUID="1e58e874-db28-4a0e-8b46-368b6d4d0960" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 01:43:29.169397 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:29.164855 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b34fc0-5b786c7bdc-8lrhc" podUID="d10ccd31-3d14-498e-bf7a-0b3034576528" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 01:43:30.246280 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:30.246239 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp"] Nov 02 01:43:30.246819 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:30.246516 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp" podUID="1371b4cc-965f-4e8c-a482-6f2e4c76282e" containerName="istio-proxy" containerID="cri-o://5faafcb73087f7b6d8a1811f8e752f1a90e2325962d4ed4041af215da5984334" gracePeriod=30 Nov 02 01:43:30.251546 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:30.251512 4427 generic.go:356] "Generic (PLEG): container finished" podID="51776e18-2d31-454f-8c33-d7fbb169c601" containerID="d54bd0fa45fe5c2c3e233a10fa99a4c2028bc5b352bf4cb393d2d486a7783358" exitCode=0 Nov 02 01:43:30.251706 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:30.251601 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w" event={"ID":"51776e18-2d31-454f-8c33-d7fbb169c601","Type":"ContainerDied","Data":"d54bd0fa45fe5c2c3e233a10fa99a4c2028bc5b352bf4cb393d2d486a7783358"} Nov 02 01:43:30.251706 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:30.251674 4427 scope.go:122] "RemoveContainer" containerID="40531b877859708237bf174c5ce262ff352f62c3d4a21cbd22017be0ad9db7ca" Nov 02 01:43:30.790143 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:30.790111 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w" Nov 02 01:43:30.824047 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:30.824007 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51776e18-2d31-454f-8c33-d7fbb169c601-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51776e18-2d31-454f-8c33-d7fbb169c601-istio-data\") pod \"51776e18-2d31-454f-8c33-d7fbb169c601\" (UID: \"51776e18-2d31-454f-8c33-d7fbb169c601\") " Nov 02 01:43:30.824234 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:30.824095 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51776e18-2d31-454f-8c33-d7fbb169c601-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51776e18-2d31-454f-8c33-d7fbb169c601-workload-socket\") pod \"51776e18-2d31-454f-8c33-d7fbb169c601\" (UID: \"51776e18-2d31-454f-8c33-d7fbb169c601\") " Nov 02 01:43:30.824234 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:30.824136 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51776e18-2d31-454f-8c33-d7fbb169c601-kube-api-access-cvfcg\" (UniqueName: \"kubernetes.io/projected/51776e18-2d31-454f-8c33-d7fbb169c601-kube-api-access-cvfcg\") pod \"51776e18-2d31-454f-8c33-d7fbb169c601\" (UID: \"51776e18-2d31-454f-8c33-d7fbb169c601\") " Nov 02 01:43:30.824234 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:30.824185 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51776e18-2d31-454f-8c33-d7fbb169c601-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51776e18-2d31-454f-8c33-d7fbb169c601-istiod-ca-cert\") pod \"51776e18-2d31-454f-8c33-d7fbb169c601\" (UID: \"51776e18-2d31-454f-8c33-d7fbb169c601\") " Nov 02 01:43:30.824234 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:30.824213 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51776e18-2d31-454f-8c33-d7fbb169c601-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51776e18-2d31-454f-8c33-d7fbb169c601-credential-socket\") pod \"51776e18-2d31-454f-8c33-d7fbb169c601\" (UID: \"51776e18-2d31-454f-8c33-d7fbb169c601\") " Nov 02 01:43:30.824523 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:30.824296 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51776e18-2d31-454f-8c33-d7fbb169c601-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51776e18-2d31-454f-8c33-d7fbb169c601-istio-podinfo\") pod \"51776e18-2d31-454f-8c33-d7fbb169c601\" (UID: \"51776e18-2d31-454f-8c33-d7fbb169c601\") " Nov 02 01:43:30.824523 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:30.824450 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51776e18-2d31-454f-8c33-d7fbb169c601-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51776e18-2d31-454f-8c33-d7fbb169c601-workload-certs\") pod \"51776e18-2d31-454f-8c33-d7fbb169c601\" (UID: \"51776e18-2d31-454f-8c33-d7fbb169c601\") " Nov 02 01:43:30.824523 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:30.824509 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51776e18-2d31-454f-8c33-d7fbb169c601-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51776e18-2d31-454f-8c33-d7fbb169c601-istio-envoy\") pod \"51776e18-2d31-454f-8c33-d7fbb169c601\" (UID: \"51776e18-2d31-454f-8c33-d7fbb169c601\") " Nov 02 01:43:30.824822 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:30.824799 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51776e18-2d31-454f-8c33-d7fbb169c601-istio-token\" (UniqueName: \"kubernetes.io/projected/51776e18-2d31-454f-8c33-d7fbb169c601-istio-token\") pod \"51776e18-2d31-454f-8c33-d7fbb169c601\" (UID: \"51776e18-2d31-454f-8c33-d7fbb169c601\") " Nov 02 01:43:30.827238 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:30.826991 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51776e18-2d31-454f-8c33-d7fbb169c601-workload-socket" pod "51776e18-2d31-454f-8c33-d7fbb169c601" (UID: "51776e18-2d31-454f-8c33-d7fbb169c601"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:30.827238 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:30.827186 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51776e18-2d31-454f-8c33-d7fbb169c601-credential-socket" pod "51776e18-2d31-454f-8c33-d7fbb169c601" (UID: "51776e18-2d31-454f-8c33-d7fbb169c601"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:30.827520 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:30.827456 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51776e18-2d31-454f-8c33-d7fbb169c601-istiod-ca-cert" pod "51776e18-2d31-454f-8c33-d7fbb169c601" (UID: "51776e18-2d31-454f-8c33-d7fbb169c601"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:43:30.827520 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:30.827514 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51776e18-2d31-454f-8c33-d7fbb169c601-workload-certs" pod "51776e18-2d31-454f-8c33-d7fbb169c601" (UID: "51776e18-2d31-454f-8c33-d7fbb169c601"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:30.828082 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:30.828059 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51776e18-2d31-454f-8c33-d7fbb169c601-istio-data" pod "51776e18-2d31-454f-8c33-d7fbb169c601" (UID: "51776e18-2d31-454f-8c33-d7fbb169c601"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:30.830016 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:30.829932 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51776e18-2d31-454f-8c33-d7fbb169c601-kube-api-access-cvfcg" pod "51776e18-2d31-454f-8c33-d7fbb169c601" (UID: "51776e18-2d31-454f-8c33-d7fbb169c601"). InnerVolumeSpecName "kube-api-access-cvfcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:43:30.833603 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:30.833556 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51776e18-2d31-454f-8c33-d7fbb169c601-istio-envoy" pod "51776e18-2d31-454f-8c33-d7fbb169c601" (UID: "51776e18-2d31-454f-8c33-d7fbb169c601"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:30.836448 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:30.836279 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51776e18-2d31-454f-8c33-d7fbb169c601-istio-podinfo" pod "51776e18-2d31-454f-8c33-d7fbb169c601" (UID: "51776e18-2d31-454f-8c33-d7fbb169c601"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:43:30.836448 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:30.836315 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51776e18-2d31-454f-8c33-d7fbb169c601-istio-token" pod "51776e18-2d31-454f-8c33-d7fbb169c601" (UID: "51776e18-2d31-454f-8c33-d7fbb169c601"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:43:30.928034 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:30.927999 4427 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51776e18-2d31-454f-8c33-d7fbb169c601-workload-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:30.928034 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:30.928031 4427 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51776e18-2d31-454f-8c33-d7fbb169c601-istio-envoy\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:30.928034 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:30.928040 4427 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51776e18-2d31-454f-8c33-d7fbb169c601-istio-token\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:30.928265 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:30.928048 4427 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51776e18-2d31-454f-8c33-d7fbb169c601-istio-data\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:30.928265 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:30.928057 4427 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51776e18-2d31-454f-8c33-d7fbb169c601-workload-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:30.928265 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:30.928065 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvfcg\" (UniqueName: \"kubernetes.io/projected/51776e18-2d31-454f-8c33-d7fbb169c601-kube-api-access-cvfcg\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:30.928265 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:30.928074 4427 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51776e18-2d31-454f-8c33-d7fbb169c601-istiod-ca-cert\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:30.928265 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:30.928082 4427 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51776e18-2d31-454f-8c33-d7fbb169c601-credential-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:30.928265 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:30.928090 4427 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51776e18-2d31-454f-8c33-d7fbb169c601-istio-podinfo\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:31.097871 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:31.097842 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8" Nov 02 01:43:31.131330 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:31.131299 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6fe33af-426d-4155-98a1-a6dfdaeead32-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6fe33af-426d-4155-98a1-a6dfdaeead32-istio-data\") pod \"a6fe33af-426d-4155-98a1-a6dfdaeead32\" (UID: \"a6fe33af-426d-4155-98a1-a6dfdaeead32\") " Nov 02 01:43:31.131503 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:31.131362 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6fe33af-426d-4155-98a1-a6dfdaeead32-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6fe33af-426d-4155-98a1-a6dfdaeead32-workload-certs\") pod \"a6fe33af-426d-4155-98a1-a6dfdaeead32\" (UID: \"a6fe33af-426d-4155-98a1-a6dfdaeead32\") " Nov 02 01:43:31.131503 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:31.131434 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6fe33af-426d-4155-98a1-a6dfdaeead32-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6fe33af-426d-4155-98a1-a6dfdaeead32-istio-envoy\") pod \"a6fe33af-426d-4155-98a1-a6dfdaeead32\" (UID: \"a6fe33af-426d-4155-98a1-a6dfdaeead32\") " Nov 02 01:43:31.131503 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:31.131497 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6fe33af-426d-4155-98a1-a6dfdaeead32-istio-token\" (UniqueName: \"kubernetes.io/projected/a6fe33af-426d-4155-98a1-a6dfdaeead32-istio-token\") pod \"a6fe33af-426d-4155-98a1-a6dfdaeead32\" (UID: \"a6fe33af-426d-4155-98a1-a6dfdaeead32\") " Nov 02 01:43:31.131617 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:31.131539 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6fe33af-426d-4155-98a1-a6dfdaeead32-kube-api-access-9z5b4\" (UniqueName: \"kubernetes.io/projected/a6fe33af-426d-4155-98a1-a6dfdaeead32-kube-api-access-9z5b4\") pod \"a6fe33af-426d-4155-98a1-a6dfdaeead32\" (UID: \"a6fe33af-426d-4155-98a1-a6dfdaeead32\") " Nov 02 01:43:31.131703 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:31.131678 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6fe33af-426d-4155-98a1-a6dfdaeead32-istio-data" pod "a6fe33af-426d-4155-98a1-a6dfdaeead32" (UID: "a6fe33af-426d-4155-98a1-a6dfdaeead32"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:31.131767 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:31.131695 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6fe33af-426d-4155-98a1-a6dfdaeead32-workload-certs" pod "a6fe33af-426d-4155-98a1-a6dfdaeead32" (UID: "a6fe33af-426d-4155-98a1-a6dfdaeead32"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:31.131767 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:31.131723 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6fe33af-426d-4155-98a1-a6dfdaeead32-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6fe33af-426d-4155-98a1-a6dfdaeead32-istiod-ca-cert\") pod \"a6fe33af-426d-4155-98a1-a6dfdaeead32\" (UID: \"a6fe33af-426d-4155-98a1-a6dfdaeead32\") " Nov 02 01:43:31.131879 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:31.131789 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6fe33af-426d-4155-98a1-a6dfdaeead32-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6fe33af-426d-4155-98a1-a6dfdaeead32-credential-socket\") pod \"a6fe33af-426d-4155-98a1-a6dfdaeead32\" (UID: \"a6fe33af-426d-4155-98a1-a6dfdaeead32\") " Nov 02 01:43:31.131879 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:31.131838 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6fe33af-426d-4155-98a1-a6dfdaeead32-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6fe33af-426d-4155-98a1-a6dfdaeead32-workload-socket\") pod \"a6fe33af-426d-4155-98a1-a6dfdaeead32\" (UID: \"a6fe33af-426d-4155-98a1-a6dfdaeead32\") " Nov 02 01:43:31.131990 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:31.131894 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a6fe33af-426d-4155-98a1-a6dfdaeead32-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6fe33af-426d-4155-98a1-a6dfdaeead32-istio-podinfo\") pod \"a6fe33af-426d-4155-98a1-a6dfdaeead32\" (UID: \"a6fe33af-426d-4155-98a1-a6dfdaeead32\") " Nov 02 01:43:31.132085 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:31.132052 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6fe33af-426d-4155-98a1-a6dfdaeead32-credential-socket" pod "a6fe33af-426d-4155-98a1-a6dfdaeead32" (UID: "a6fe33af-426d-4155-98a1-a6dfdaeead32"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:31.132158 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:31.132135 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6fe33af-426d-4155-98a1-a6dfdaeead32-workload-socket" pod "a6fe33af-426d-4155-98a1-a6dfdaeead32" (UID: "a6fe33af-426d-4155-98a1-a6dfdaeead32"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:31.132213 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:31.132177 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6fe33af-426d-4155-98a1-a6dfdaeead32-istiod-ca-cert" pod "a6fe33af-426d-4155-98a1-a6dfdaeead32" (UID: "a6fe33af-426d-4155-98a1-a6dfdaeead32"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:43:31.134764 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:31.134735 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6fe33af-426d-4155-98a1-a6dfdaeead32-istio-envoy" pod "a6fe33af-426d-4155-98a1-a6dfdaeead32" (UID: "a6fe33af-426d-4155-98a1-a6dfdaeead32"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:31.134887 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:31.134827 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a6fe33af-426d-4155-98a1-a6dfdaeead32-istio-podinfo" pod "a6fe33af-426d-4155-98a1-a6dfdaeead32" (UID: "a6fe33af-426d-4155-98a1-a6dfdaeead32"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:43:31.134934 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:31.134875 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6fe33af-426d-4155-98a1-a6dfdaeead32-istio-token" pod "a6fe33af-426d-4155-98a1-a6dfdaeead32" (UID: "a6fe33af-426d-4155-98a1-a6dfdaeead32"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:43:31.135258 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:31.135225 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6fe33af-426d-4155-98a1-a6dfdaeead32-kube-api-access-9z5b4" pod "a6fe33af-426d-4155-98a1-a6dfdaeead32" (UID: "a6fe33af-426d-4155-98a1-a6dfdaeead32"). InnerVolumeSpecName "kube-api-access-9z5b4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:43:31.135758 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:31.135738 4427 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6fe33af-426d-4155-98a1-a6dfdaeead32-istio-data\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:31.135842 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:31.135760 4427 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6fe33af-426d-4155-98a1-a6dfdaeead32-workload-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:31.135842 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:31.135771 4427 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6fe33af-426d-4155-98a1-a6dfdaeead32-istio-envoy\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:31.135842 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:31.135779 4427 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6fe33af-426d-4155-98a1-a6dfdaeead32-istio-token\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:31.135842 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:31.135788 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9z5b4\" (UniqueName: \"kubernetes.io/projected/a6fe33af-426d-4155-98a1-a6dfdaeead32-kube-api-access-9z5b4\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:31.135842 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:31.135797 4427 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6fe33af-426d-4155-98a1-a6dfdaeead32-istiod-ca-cert\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:31.135842 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:31.135806 4427 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6fe33af-426d-4155-98a1-a6dfdaeead32-credential-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:31.135842 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:31.135814 4427 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6fe33af-426d-4155-98a1-a6dfdaeead32-workload-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:31.135842 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:31.135822 4427 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6fe33af-426d-4155-98a1-a6dfdaeead32-istio-podinfo\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:31.187827 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:31.187781 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-addad9-7579959b99-l926m" podUID="3d9b8d38-b3ce-4556-8580-93cfd36b2d2a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 01:43:31.263765 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:31.263662 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w" Nov 02 01:43:31.263765 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:31.263700 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w" event={"ID":"51776e18-2d31-454f-8c33-d7fbb169c601","Type":"ContainerDied","Data":"c6fa163efa9f23a67ed4f6f8384d79e81973c1f93fee1da481c0db9fe99d998d"} Nov 02 01:43:31.264366 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:31.263793 4427 scope.go:122] "RemoveContainer" containerID="d54bd0fa45fe5c2c3e233a10fa99a4c2028bc5b352bf4cb393d2d486a7783358" Nov 02 01:43:31.272201 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:31.272142 4427 generic.go:356] "Generic (PLEG): container finished" podID="a6fe33af-426d-4155-98a1-a6dfdaeead32" containerID="a0db62a8741afd6232db998ff821a91006a2990a18326b967f8a037f4bd7e8d4" exitCode=0 Nov 02 01:43:31.272201 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:31.272191 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8" event={"ID":"a6fe33af-426d-4155-98a1-a6dfdaeead32","Type":"ContainerDied","Data":"a0db62a8741afd6232db998ff821a91006a2990a18326b967f8a037f4bd7e8d4"} Nov 02 01:43:31.272495 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:31.272198 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8" Nov 02 01:43:31.275247 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:31.272245 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8" event={"ID":"a6fe33af-426d-4155-98a1-a6dfdaeead32","Type":"ContainerDied","Data":"9a6a59f6e21034b6bd87f9eb34e9437a3baffa29e9158c2886aba587aae25315"} Nov 02 01:43:31.284687 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:31.284661 4427 scope.go:122] "RemoveContainer" containerID="a0db62a8741afd6232db998ff821a91006a2990a18326b967f8a037f4bd7e8d4" Nov 02 01:43:31.298028 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:31.297998 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w"] Nov 02 01:43:31.301347 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:31.301327 4427 scope.go:122] "RemoveContainer" containerID="a0db62a8741afd6232db998ff821a91006a2990a18326b967f8a037f4bd7e8d4" Nov 02 01:43:31.301714 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:43:31.301693 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0db62a8741afd6232db998ff821a91006a2990a18326b967f8a037f4bd7e8d4\": container with ID starting with a0db62a8741afd6232db998ff821a91006a2990a18326b967f8a037f4bd7e8d4 not found: ID does not exist" containerID="a0db62a8741afd6232db998ff821a91006a2990a18326b967f8a037f4bd7e8d4" Nov 02 01:43:31.301798 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:31.301725 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0db62a8741afd6232db998ff821a91006a2990a18326b967f8a037f4bd7e8d4"} err="failed to get container status \"a0db62a8741afd6232db998ff821a91006a2990a18326b967f8a037f4bd7e8d4\": rpc error: code = NotFound desc = could not find container \"a0db62a8741afd6232db998ff821a91006a2990a18326b967f8a037f4bd7e8d4\": container with ID starting with a0db62a8741afd6232db998ff821a91006a2990a18326b967f8a037f4bd7e8d4 not found: ID does not exist" Nov 02 01:43:31.304532 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:31.304506 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-14b63f-openshift-default-6d59f89685-fg47w"] Nov 02 01:43:31.318591 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:31.317351 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8"] Nov 02 01:43:31.325296 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:31.325259 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-e412d5-openshift-default-58c567cbc9-drnm8"] Nov 02 01:43:31.536917 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:31.536840 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51776e18-2d31-454f-8c33-d7fbb169c601" path="/var/lib/kubelet/pods/51776e18-2d31-454f-8c33-d7fbb169c601/volumes" Nov 02 01:43:31.537510 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:31.537495 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6fe33af-426d-4155-98a1-a6dfdaeead32" path="/var/lib/kubelet/pods/a6fe33af-426d-4155-98a1-a6dfdaeead32/volumes" Nov 02 01:43:31.586169 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:43:31.586133 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:43:32.087546 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:32.087505 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b"] Nov 02 01:43:32.092682 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:32.092651 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs"] Nov 02 01:43:34.981489 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:34.981461 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-addad9-7579959b99-l926m" Nov 02 01:43:35.116682 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:35.116652 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d9b8d38-b3ce-4556-8580-93cfd36b2d2a-kube-api-access-4lwzv\" (UniqueName: \"kubernetes.io/projected/3d9b8d38-b3ce-4556-8580-93cfd36b2d2a-kube-api-access-4lwzv\") pod \"3d9b8d38-b3ce-4556-8580-93cfd36b2d2a\" (UID: \"3d9b8d38-b3ce-4556-8580-93cfd36b2d2a\") " Nov 02 01:43:35.116891 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:35.116796 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d9b8d38-b3ce-4556-8580-93cfd36b2d2a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d9b8d38-b3ce-4556-8580-93cfd36b2d2a-plugins-config-volume\") pod \"3d9b8d38-b3ce-4556-8580-93cfd36b2d2a\" (UID: \"3d9b8d38-b3ce-4556-8580-93cfd36b2d2a\") " Nov 02 01:43:35.117246 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:35.117214 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d9b8d38-b3ce-4556-8580-93cfd36b2d2a-plugins-config-volume" pod "3d9b8d38-b3ce-4556-8580-93cfd36b2d2a" (UID: "3d9b8d38-b3ce-4556-8580-93cfd36b2d2a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:43:35.117707 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:35.117689 4427 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d9b8d38-b3ce-4556-8580-93cfd36b2d2a-plugins-config-volume\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:35.119070 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:35.119049 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d9b8d38-b3ce-4556-8580-93cfd36b2d2a-kube-api-access-4lwzv" pod "3d9b8d38-b3ce-4556-8580-93cfd36b2d2a" (UID: "3d9b8d38-b3ce-4556-8580-93cfd36b2d2a"). InnerVolumeSpecName "kube-api-access-4lwzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:43:35.219483 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:35.219404 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lwzv\" (UniqueName: \"kubernetes.io/projected/3d9b8d38-b3ce-4556-8580-93cfd36b2d2a-kube-api-access-4lwzv\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:35.356475 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:35.356441 4427 generic.go:356] "Generic (PLEG): container finished" podID="3d9b8d38-b3ce-4556-8580-93cfd36b2d2a" containerID="ef8e82b2a4a793ac0975e87e30e9b105c4d4e8ed64ae72f586943d4999e84141" exitCode=0 Nov 02 01:43:35.356656 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:35.356510 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-addad9-7579959b99-l926m" Nov 02 01:43:35.356656 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:35.356525 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-addad9-7579959b99-l926m" event={"ID":"3d9b8d38-b3ce-4556-8580-93cfd36b2d2a","Type":"ContainerDied","Data":"ef8e82b2a4a793ac0975e87e30e9b105c4d4e8ed64ae72f586943d4999e84141"} Nov 02 01:43:35.356656 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:35.356583 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-addad9-7579959b99-l926m" event={"ID":"3d9b8d38-b3ce-4556-8580-93cfd36b2d2a","Type":"ContainerDied","Data":"625b5c066dcf4e45ff905c05ee2a0e01b80647620d658efa5ecfb693e21b7a19"} Nov 02 01:43:35.356656 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:35.356615 4427 scope.go:122] "RemoveContainer" containerID="ef8e82b2a4a793ac0975e87e30e9b105c4d4e8ed64ae72f586943d4999e84141" Nov 02 01:43:35.378051 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:35.378024 4427 scope.go:122] "RemoveContainer" containerID="ef8e82b2a4a793ac0975e87e30e9b105c4d4e8ed64ae72f586943d4999e84141" Nov 02 01:43:35.378420 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:43:35.378390 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef8e82b2a4a793ac0975e87e30e9b105c4d4e8ed64ae72f586943d4999e84141\": container with ID starting with ef8e82b2a4a793ac0975e87e30e9b105c4d4e8ed64ae72f586943d4999e84141 not found: ID does not exist" containerID="ef8e82b2a4a793ac0975e87e30e9b105c4d4e8ed64ae72f586943d4999e84141" Nov 02 01:43:35.378530 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:35.378424 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef8e82b2a4a793ac0975e87e30e9b105c4d4e8ed64ae72f586943d4999e84141"} err="failed to get container status \"ef8e82b2a4a793ac0975e87e30e9b105c4d4e8ed64ae72f586943d4999e84141\": rpc error: code = NotFound desc = could not find container \"ef8e82b2a4a793ac0975e87e30e9b105c4d4e8ed64ae72f586943d4999e84141\": container with ID starting with ef8e82b2a4a793ac0975e87e30e9b105c4d4e8ed64ae72f586943d4999e84141 not found: ID does not exist" Nov 02 01:43:35.393018 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:35.392987 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-addad9-7579959b99-l926m"] Nov 02 01:43:35.402493 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:35.402461 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-addad9-7579959b99-l926m"] Nov 02 01:43:35.923439 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:35.923067 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp" Nov 02 01:43:35.932760 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:35.925097 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d9b8d38-b3ce-4556-8580-93cfd36b2d2a" path="/var/lib/kubelet/pods/3d9b8d38-b3ce-4556-8580-93cfd36b2d2a/volumes" Nov 02 01:43:36.079649 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:36.079612 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1371b4cc-965f-4e8c-a482-6f2e4c76282e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1371b4cc-965f-4e8c-a482-6f2e4c76282e-istiod-ca-cert\") pod \"1371b4cc-965f-4e8c-a482-6f2e4c76282e\" (UID: \"1371b4cc-965f-4e8c-a482-6f2e4c76282e\") " Nov 02 01:43:36.080077 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:36.079723 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1371b4cc-965f-4e8c-a482-6f2e4c76282e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1371b4cc-965f-4e8c-a482-6f2e4c76282e-istio-data\") pod \"1371b4cc-965f-4e8c-a482-6f2e4c76282e\" (UID: \"1371b4cc-965f-4e8c-a482-6f2e4c76282e\") " Nov 02 01:43:36.080077 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:36.079756 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1371b4cc-965f-4e8c-a482-6f2e4c76282e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1371b4cc-965f-4e8c-a482-6f2e4c76282e-workload-certs\") pod \"1371b4cc-965f-4e8c-a482-6f2e4c76282e\" (UID: \"1371b4cc-965f-4e8c-a482-6f2e4c76282e\") " Nov 02 01:43:36.080077 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:36.079888 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1371b4cc-965f-4e8c-a482-6f2e4c76282e-kube-api-access-bntfl\" (UniqueName: \"kubernetes.io/projected/1371b4cc-965f-4e8c-a482-6f2e4c76282e-kube-api-access-bntfl\") pod \"1371b4cc-965f-4e8c-a482-6f2e4c76282e\" (UID: \"1371b4cc-965f-4e8c-a482-6f2e4c76282e\") " Nov 02 01:43:36.080077 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:36.079952 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1371b4cc-965f-4e8c-a482-6f2e4c76282e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1371b4cc-965f-4e8c-a482-6f2e4c76282e-istio-envoy\") pod \"1371b4cc-965f-4e8c-a482-6f2e4c76282e\" (UID: \"1371b4cc-965f-4e8c-a482-6f2e4c76282e\") " Nov 02 01:43:36.080077 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:36.079999 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1371b4cc-965f-4e8c-a482-6f2e4c76282e-istiod-ca-cert" pod "1371b4cc-965f-4e8c-a482-6f2e4c76282e" (UID: "1371b4cc-965f-4e8c-a482-6f2e4c76282e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:43:36.080077 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:36.080061 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1371b4cc-965f-4e8c-a482-6f2e4c76282e-istio-data" pod "1371b4cc-965f-4e8c-a482-6f2e4c76282e" (UID: "1371b4cc-965f-4e8c-a482-6f2e4c76282e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:36.080513 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:36.080122 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1371b4cc-965f-4e8c-a482-6f2e4c76282e-workload-certs" pod "1371b4cc-965f-4e8c-a482-6f2e4c76282e" (UID: "1371b4cc-965f-4e8c-a482-6f2e4c76282e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:36.080513 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:36.080135 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1371b4cc-965f-4e8c-a482-6f2e4c76282e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1371b4cc-965f-4e8c-a482-6f2e4c76282e-istio-podinfo\") pod \"1371b4cc-965f-4e8c-a482-6f2e4c76282e\" (UID: \"1371b4cc-965f-4e8c-a482-6f2e4c76282e\") " Nov 02 01:43:36.080513 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:36.080200 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1371b4cc-965f-4e8c-a482-6f2e4c76282e-istio-token\" (UniqueName: \"kubernetes.io/projected/1371b4cc-965f-4e8c-a482-6f2e4c76282e-istio-token\") pod \"1371b4cc-965f-4e8c-a482-6f2e4c76282e\" (UID: \"1371b4cc-965f-4e8c-a482-6f2e4c76282e\") " Nov 02 01:43:36.080513 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:36.080258 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1371b4cc-965f-4e8c-a482-6f2e4c76282e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1371b4cc-965f-4e8c-a482-6f2e4c76282e-credential-socket\") pod \"1371b4cc-965f-4e8c-a482-6f2e4c76282e\" (UID: \"1371b4cc-965f-4e8c-a482-6f2e4c76282e\") " Nov 02 01:43:36.080513 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:36.080365 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1371b4cc-965f-4e8c-a482-6f2e4c76282e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1371b4cc-965f-4e8c-a482-6f2e4c76282e-workload-socket\") pod \"1371b4cc-965f-4e8c-a482-6f2e4c76282e\" (UID: \"1371b4cc-965f-4e8c-a482-6f2e4c76282e\") " Nov 02 01:43:36.080697 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:36.080581 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1371b4cc-965f-4e8c-a482-6f2e4c76282e-credential-socket" pod "1371b4cc-965f-4e8c-a482-6f2e4c76282e" (UID: "1371b4cc-965f-4e8c-a482-6f2e4c76282e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:36.080697 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:36.080605 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1371b4cc-965f-4e8c-a482-6f2e4c76282e-workload-socket" pod "1371b4cc-965f-4e8c-a482-6f2e4c76282e" (UID: "1371b4cc-965f-4e8c-a482-6f2e4c76282e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:36.081807 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:36.081777 4427 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1371b4cc-965f-4e8c-a482-6f2e4c76282e-credential-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:36.081976 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:36.081812 4427 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1371b4cc-965f-4e8c-a482-6f2e4c76282e-workload-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:36.081976 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:36.081829 4427 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1371b4cc-965f-4e8c-a482-6f2e4c76282e-istiod-ca-cert\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:36.081976 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:36.081843 4427 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1371b4cc-965f-4e8c-a482-6f2e4c76282e-istio-data\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:36.081976 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:36.081857 4427 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1371b4cc-965f-4e8c-a482-6f2e4c76282e-workload-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:36.082630 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:36.082595 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1371b4cc-965f-4e8c-a482-6f2e4c76282e-istio-envoy" pod "1371b4cc-965f-4e8c-a482-6f2e4c76282e" (UID: "1371b4cc-965f-4e8c-a482-6f2e4c76282e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:36.082746 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:36.082675 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1371b4cc-965f-4e8c-a482-6f2e4c76282e-kube-api-access-bntfl" pod "1371b4cc-965f-4e8c-a482-6f2e4c76282e" (UID: "1371b4cc-965f-4e8c-a482-6f2e4c76282e"). InnerVolumeSpecName "kube-api-access-bntfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:43:36.082746 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:36.082698 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1371b4cc-965f-4e8c-a482-6f2e4c76282e-istio-token" pod "1371b4cc-965f-4e8c-a482-6f2e4c76282e" (UID: "1371b4cc-965f-4e8c-a482-6f2e4c76282e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:43:36.082887 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:36.082869 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1371b4cc-965f-4e8c-a482-6f2e4c76282e-istio-podinfo" pod "1371b4cc-965f-4e8c-a482-6f2e4c76282e" (UID: "1371b4cc-965f-4e8c-a482-6f2e4c76282e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:43:36.184102 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:36.183999 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bntfl\" (UniqueName: \"kubernetes.io/projected/1371b4cc-965f-4e8c-a482-6f2e4c76282e-kube-api-access-bntfl\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:36.184102 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:36.184040 4427 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1371b4cc-965f-4e8c-a482-6f2e4c76282e-istio-envoy\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:36.184102 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:36.184054 4427 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1371b4cc-965f-4e8c-a482-6f2e4c76282e-istio-podinfo\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:36.184102 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:36.184066 4427 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1371b4cc-965f-4e8c-a482-6f2e4c76282e-istio-token\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:36.367986 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:36.367951 4427 generic.go:356] "Generic (PLEG): container finished" podID="1371b4cc-965f-4e8c-a482-6f2e4c76282e" containerID="5faafcb73087f7b6d8a1811f8e752f1a90e2325962d4ed4041af215da5984334" exitCode=0 Nov 02 01:43:36.368167 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:36.368026 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp" Nov 02 01:43:36.368167 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:36.368031 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp" event={"ID":"1371b4cc-965f-4e8c-a482-6f2e4c76282e","Type":"ContainerDied","Data":"5faafcb73087f7b6d8a1811f8e752f1a90e2325962d4ed4041af215da5984334"} Nov 02 01:43:36.368167 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:36.368112 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp" event={"ID":"1371b4cc-965f-4e8c-a482-6f2e4c76282e","Type":"ContainerDied","Data":"dbca344df7ba3a449d9679a4ab32dc7b5602b20e3031bf585baff6d602a9358e"} Nov 02 01:43:36.368167 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:36.368135 4427 scope.go:122] "RemoveContainer" containerID="5faafcb73087f7b6d8a1811f8e752f1a90e2325962d4ed4041af215da5984334" Nov 02 01:43:36.381691 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:36.381671 4427 scope.go:122] "RemoveContainer" containerID="5faafcb73087f7b6d8a1811f8e752f1a90e2325962d4ed4041af215da5984334" Nov 02 01:43:36.381991 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:43:36.381965 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5faafcb73087f7b6d8a1811f8e752f1a90e2325962d4ed4041af215da5984334\": container with ID starting with 5faafcb73087f7b6d8a1811f8e752f1a90e2325962d4ed4041af215da5984334 not found: ID does not exist" containerID="5faafcb73087f7b6d8a1811f8e752f1a90e2325962d4ed4041af215da5984334" Nov 02 01:43:36.382165 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:36.381995 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5faafcb73087f7b6d8a1811f8e752f1a90e2325962d4ed4041af215da5984334"} err="failed to get container status \"5faafcb73087f7b6d8a1811f8e752f1a90e2325962d4ed4041af215da5984334\": rpc error: code = NotFound desc = could not find container \"5faafcb73087f7b6d8a1811f8e752f1a90e2325962d4ed4041af215da5984334\": container with ID starting with 5faafcb73087f7b6d8a1811f8e752f1a90e2325962d4ed4041af215da5984334 not found: ID does not exist" Nov 02 01:43:36.397276 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:36.397234 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp"] Nov 02 01:43:36.402145 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:36.402097 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-addad9-openshift-default-69f48df95c-sxfmp"] Nov 02 01:43:37.397745 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:37.397707 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l"] Nov 02 01:43:37.398234 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:37.397996 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l" podUID="e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0" containerName="istio-proxy" containerID="cri-o://5ac18ab0bce64b05e7e0e8b3a8a9c47fd50e9a7cf9d7e587aef100e5b71eeb5a" gracePeriod=30 Nov 02 01:43:37.539313 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:37.539281 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1371b4cc-965f-4e8c-a482-6f2e4c76282e" path="/var/lib/kubelet/pods/1371b4cc-965f-4e8c-a482-6f2e4c76282e/volumes" Nov 02 01:43:39.111581 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:39.111544 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b34fc0-5b786c7bdc-8lrhc" podUID="d10ccd31-3d14-498e-bf7a-0b3034576528" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 01:43:39.111978 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:39.111668 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-b34fc0-6cd9f9c5fc-9n8zq" podUID="1e58e874-db28-4a0e-8b46-368b6d4d0960" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 01:43:40.991292 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:40.991243 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2589b2-5b4dc6cc9f-nmtkc"] Nov 02 01:43:42.752539 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:42.752507 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l" Nov 02 01:43:42.776399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:42.773931 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0-credential-socket\") pod \"e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0\" (UID: \"e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0\") " Nov 02 01:43:42.776399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:42.774058 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0-kube-api-access-7zchs\" (UniqueName: \"kubernetes.io/projected/e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0-kube-api-access-7zchs\") pod \"e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0\" (UID: \"e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0\") " Nov 02 01:43:42.776399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:42.774116 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0-workload-socket\") pod \"e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0\" (UID: \"e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0\") " Nov 02 01:43:42.776399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:42.774176 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0-istio-podinfo\") pod \"e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0\" (UID: \"e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0\") " Nov 02 01:43:42.776399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:42.774223 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0-istio-envoy\") pod \"e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0\" (UID: \"e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0\") " Nov 02 01:43:42.776399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:42.774276 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0-istio-data\") pod \"e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0\" (UID: \"e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0\") " Nov 02 01:43:42.776399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:42.774360 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0-istiod-ca-cert\") pod \"e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0\" (UID: \"e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0\") " Nov 02 01:43:42.776399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:42.774456 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0-istio-token\" (UniqueName: \"kubernetes.io/projected/e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0-istio-token\") pod \"e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0\" (UID: \"e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0\") " Nov 02 01:43:42.776399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:42.774496 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0-workload-certs\") pod \"e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0\" (UID: \"e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0\") " Nov 02 01:43:42.776399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:42.775690 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0-workload-certs" pod "e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0" (UID: "e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:42.776399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:42.775948 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0-credential-socket" pod "e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0" (UID: "e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:42.779708 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:42.779364 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0-kube-api-access-7zchs" pod "e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0" (UID: "e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0"). InnerVolumeSpecName "kube-api-access-7zchs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:43:42.779708 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:42.779682 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0-workload-socket" pod "e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0" (UID: "e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:42.781279 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:42.781243 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0-istiod-ca-cert" pod "e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0" (UID: "e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:43:42.781545 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:42.781522 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0-istio-data" pod "e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0" (UID: "e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:42.782698 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:42.782662 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0-istio-podinfo" pod "e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0" (UID: "e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:43:42.785139 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:42.785111 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0-istio-envoy" pod "e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0" (UID: "e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:42.804449 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:42.792425 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0-istio-token" pod "e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0" (UID: "e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:43:42.877647 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:42.877602 4427 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0-workload-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:42.877647 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:42.877636 4427 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0-istio-podinfo\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:42.877647 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:42.877652 4427 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0-istio-envoy\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:42.877903 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:42.877664 4427 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0-istio-data\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:42.877903 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:42.877676 4427 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0-istiod-ca-cert\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:42.877903 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:42.877687 4427 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0-istio-token\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:42.877903 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:42.877699 4427 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0-workload-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:42.877903 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:42.877710 4427 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0-credential-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:42.877903 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:42.877722 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zchs\" (UniqueName: \"kubernetes.io/projected/e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0-kube-api-access-7zchs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:43.507894 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:43.507839 4427 generic.go:356] "Generic (PLEG): container finished" podID="e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0" containerID="5ac18ab0bce64b05e7e0e8b3a8a9c47fd50e9a7cf9d7e587aef100e5b71eeb5a" exitCode=0 Nov 02 01:43:43.508098 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:43.508068 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l" event={"ID":"e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0","Type":"ContainerDied","Data":"5ac18ab0bce64b05e7e0e8b3a8a9c47fd50e9a7cf9d7e587aef100e5b71eeb5a"} Nov 02 01:43:43.508206 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:43.508182 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l" event={"ID":"e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0","Type":"ContainerDied","Data":"a8e28c7f3bf765c1c5bfe29c6317dca8f4b80825096b35d99f7bc2063e1fb19c"} Nov 02 01:43:43.508301 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:43.508252 4427 scope.go:122] "RemoveContainer" containerID="5ac18ab0bce64b05e7e0e8b3a8a9c47fd50e9a7cf9d7e587aef100e5b71eeb5a" Nov 02 01:43:43.508624 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:43.508601 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l" Nov 02 01:43:43.525110 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:43.525083 4427 scope.go:122] "RemoveContainer" containerID="db7574e374b5d799907ecd4f3b953b8d7c2afccfa515149ab1a326ea05191cae" Nov 02 01:43:43.538009 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:43.537980 4427 scope.go:122] "RemoveContainer" containerID="5ac18ab0bce64b05e7e0e8b3a8a9c47fd50e9a7cf9d7e587aef100e5b71eeb5a" Nov 02 01:43:43.538305 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:43:43.538283 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ac18ab0bce64b05e7e0e8b3a8a9c47fd50e9a7cf9d7e587aef100e5b71eeb5a\": container with ID starting with 5ac18ab0bce64b05e7e0e8b3a8a9c47fd50e9a7cf9d7e587aef100e5b71eeb5a not found: ID does not exist" containerID="5ac18ab0bce64b05e7e0e8b3a8a9c47fd50e9a7cf9d7e587aef100e5b71eeb5a" Nov 02 01:43:43.538422 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:43.538315 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ac18ab0bce64b05e7e0e8b3a8a9c47fd50e9a7cf9d7e587aef100e5b71eeb5a"} err="failed to get container status \"5ac18ab0bce64b05e7e0e8b3a8a9c47fd50e9a7cf9d7e587aef100e5b71eeb5a\": rpc error: code = NotFound desc = could not find container \"5ac18ab0bce64b05e7e0e8b3a8a9c47fd50e9a7cf9d7e587aef100e5b71eeb5a\": container with ID starting with 5ac18ab0bce64b05e7e0e8b3a8a9c47fd50e9a7cf9d7e587aef100e5b71eeb5a not found: ID does not exist" Nov 02 01:43:43.538422 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:43.538333 4427 scope.go:122] "RemoveContainer" containerID="db7574e374b5d799907ecd4f3b953b8d7c2afccfa515149ab1a326ea05191cae" Nov 02 01:43:43.538706 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:43:43.538676 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db7574e374b5d799907ecd4f3b953b8d7c2afccfa515149ab1a326ea05191cae\": container with ID starting with db7574e374b5d799907ecd4f3b953b8d7c2afccfa515149ab1a326ea05191cae not found: ID does not exist" containerID="db7574e374b5d799907ecd4f3b953b8d7c2afccfa515149ab1a326ea05191cae" Nov 02 01:43:43.538834 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:43.538715 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db7574e374b5d799907ecd4f3b953b8d7c2afccfa515149ab1a326ea05191cae"} err="failed to get container status \"db7574e374b5d799907ecd4f3b953b8d7c2afccfa515149ab1a326ea05191cae\": rpc error: code = NotFound desc = could not find container \"db7574e374b5d799907ecd4f3b953b8d7c2afccfa515149ab1a326ea05191cae\": container with ID starting with db7574e374b5d799907ecd4f3b953b8d7c2afccfa515149ab1a326ea05191cae not found: ID does not exist" Nov 02 01:43:43.541595 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:43.541569 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l"] Nov 02 01:43:43.546881 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:43.546857 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-0bbeda-openshift-default-5c79f6b6c-ll89l"] Nov 02 01:43:45.536927 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:45.536892 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0" path="/var/lib/kubelet/pods/e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0/volumes" Nov 02 01:43:46.048456 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:46.036232 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2589b2-5b4dc6cc9f-nmtkc" podUID="a097a1f1-7f03-479d-a65a-90f8e542d06b" containerName="payload-pre-processing" containerID="cri-o://b99b75d2d5d0e26802f6fa73863ea110cfe140e58ceca80294d507733ca008e8" gracePeriod=25 Nov 02 01:43:47.112425 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.112385 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" podUID="ea61971c-5a72-4609-b5a0-58a74b324bb7" containerName="main" containerID="cri-o://e8a579faafe103c3a9a2216d88d2dcba9caa4e82a8e7784965ef4d349389d44c" gracePeriod=45 Nov 02 01:43:47.117237 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.117197 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" podUID="c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6" containerName="main" containerID="cri-o://5ad14d31ea4019e07ab69f7baf7c92eeb8d8e042e2f0fd5cf41634783065770a" gracePeriod=45 Nov 02 01:43:47.558664 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.558623 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" Nov 02 01:43:47.562112 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.562084 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" Nov 02 01:43:47.565442 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.565413 4427 generic.go:356] "Generic (PLEG): container finished" podID="1e58e874-db28-4a0e-8b46-368b6d4d0960" containerID="d65fa69dbf28c35abecfbb9bcfc5765dc93643496377c7228266dd0edfc22ec0" exitCode=0 Nov 02 01:43:47.565971 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.565946 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b34fc0-6cd9f9c5fc-9n8zq" event={"ID":"1e58e874-db28-4a0e-8b46-368b6d4d0960","Type":"ContainerDied","Data":"d65fa69dbf28c35abecfbb9bcfc5765dc93643496377c7228266dd0edfc22ec0"} Nov 02 01:43:47.569013 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.568985 4427 generic.go:356] "Generic (PLEG): container finished" podID="ea61971c-5a72-4609-b5a0-58a74b324bb7" containerID="e8a579faafe103c3a9a2216d88d2dcba9caa4e82a8e7784965ef4d349389d44c" exitCode=0 Nov 02 01:43:47.569339 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.569315 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" event={"ID":"ea61971c-5a72-4609-b5a0-58a74b324bb7","Type":"ContainerDied","Data":"e8a579faafe103c3a9a2216d88d2dcba9caa4e82a8e7784965ef4d349389d44c"} Nov 02 01:43:47.569502 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.569485 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" event={"ID":"ea61971c-5a72-4609-b5a0-58a74b324bb7","Type":"ContainerDied","Data":"6ec71cadf23ae919ed1c23ca0d12d32a998ffefd8b6f4742db6b83fdab86e10d"} Nov 02 01:43:47.569614 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.569602 4427 scope.go:122] "RemoveContainer" containerID="e8a579faafe103c3a9a2216d88d2dcba9caa4e82a8e7784965ef4d349389d44c" Nov 02 01:43:47.569825 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.569810 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b" Nov 02 01:43:47.575168 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.575144 4427 generic.go:356] "Generic (PLEG): container finished" podID="c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6" containerID="5ad14d31ea4019e07ab69f7baf7c92eeb8d8e042e2f0fd5cf41634783065770a" exitCode=0 Nov 02 01:43:47.575275 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.575250 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" event={"ID":"c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6","Type":"ContainerDied","Data":"5ad14d31ea4019e07ab69f7baf7c92eeb8d8e042e2f0fd5cf41634783065770a"} Nov 02 01:43:47.575316 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.575300 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" event={"ID":"c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6","Type":"ContainerDied","Data":"380c736113518d7386efb87845d883df672246590fe211f6f764d5ac4c919a83"} Nov 02 01:43:47.575424 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.575396 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" Nov 02 01:43:47.578488 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.578275 4427 generic.go:356] "Generic (PLEG): container finished" podID="d10ccd31-3d14-498e-bf7a-0b3034576528" containerID="ba8d5bfd013544907336604267b6db353fb410700c119045e616171898fb616a" exitCode=0 Nov 02 01:43:47.578488 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.578356 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b34fc0-5b786c7bdc-8lrhc" event={"ID":"d10ccd31-3d14-498e-bf7a-0b3034576528","Type":"ContainerDied","Data":"ba8d5bfd013544907336604267b6db353fb410700c119045e616171898fb616a"} Nov 02 01:43:47.621222 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.621195 4427 scope.go:122] "RemoveContainer" containerID="e8a579faafe103c3a9a2216d88d2dcba9caa4e82a8e7784965ef4d349389d44c" Nov 02 01:43:47.622084 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:43:47.621941 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8a579faafe103c3a9a2216d88d2dcba9caa4e82a8e7784965ef4d349389d44c\": container with ID starting with e8a579faafe103c3a9a2216d88d2dcba9caa4e82a8e7784965ef4d349389d44c not found: ID does not exist" containerID="e8a579faafe103c3a9a2216d88d2dcba9caa4e82a8e7784965ef4d349389d44c" Nov 02 01:43:47.622084 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.621979 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8a579faafe103c3a9a2216d88d2dcba9caa4e82a8e7784965ef4d349389d44c"} err="failed to get container status \"e8a579faafe103c3a9a2216d88d2dcba9caa4e82a8e7784965ef4d349389d44c\": rpc error: code = NotFound desc = could not find container \"e8a579faafe103c3a9a2216d88d2dcba9caa4e82a8e7784965ef4d349389d44c\": container with ID starting with e8a579faafe103c3a9a2216d88d2dcba9caa4e82a8e7784965ef4d349389d44c not found: ID does not exist" Nov 02 01:43:47.622084 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.622001 4427 scope.go:122] "RemoveContainer" containerID="5ad14d31ea4019e07ab69f7baf7c92eeb8d8e042e2f0fd5cf41634783065770a" Nov 02 01:43:47.671220 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.671165 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4"] Nov 02 01:43:47.672422 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.671829 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4" podUID="c3d179f1-c7e0-4340-a286-2efbbc44ed53" containerName="istio-proxy" containerID="cri-o://a7ff2e236648abc871bf55a084d15da6ae70b7df221bc21f7fa2e80d0df88c64" gracePeriod=30 Nov 02 01:43:47.679468 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.676931 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea61971c-5a72-4609-b5a0-58a74b324bb7-home\" (UniqueName: \"kubernetes.io/empty-dir/ea61971c-5a72-4609-b5a0-58a74b324bb7-home\") pod \"ea61971c-5a72-4609-b5a0-58a74b324bb7\" (UID: \"ea61971c-5a72-4609-b5a0-58a74b324bb7\") " Nov 02 01:43:47.679468 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.677015 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6-tls-certs\" (UniqueName: \"kubernetes.io/secret/c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6-tls-certs\") pod \"c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6\" (UID: \"c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6\") " Nov 02 01:43:47.679468 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.677059 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6-dshm\") pod \"c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6\" (UID: \"c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6\") " Nov 02 01:43:47.679468 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.677109 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6-tmp-dir\") pod \"c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6\" (UID: \"c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6\") " Nov 02 01:43:47.679468 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.677193 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea61971c-5a72-4609-b5a0-58a74b324bb7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea61971c-5a72-4609-b5a0-58a74b324bb7-model-cache\") pod \"ea61971c-5a72-4609-b5a0-58a74b324bb7\" (UID: \"ea61971c-5a72-4609-b5a0-58a74b324bb7\") " Nov 02 01:43:47.679468 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.677299 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea61971c-5a72-4609-b5a0-58a74b324bb7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea61971c-5a72-4609-b5a0-58a74b324bb7-dshm\") pod \"ea61971c-5a72-4609-b5a0-58a74b324bb7\" (UID: \"ea61971c-5a72-4609-b5a0-58a74b324bb7\") " Nov 02 01:43:47.679468 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.677426 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6-kube-api-access-vz4fv\" (UniqueName: \"kubernetes.io/projected/c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6-kube-api-access-vz4fv\") pod \"c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6\" (UID: \"c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6\") " Nov 02 01:43:47.679468 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.677469 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6-home\" (UniqueName: \"kubernetes.io/empty-dir/c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6-home\") pod \"c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6\" (UID: \"c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6\") " Nov 02 01:43:47.679468 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.677547 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6-model-cache\") pod \"c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6\" (UID: \"c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6\") " Nov 02 01:43:47.679468 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.677770 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea61971c-5a72-4609-b5a0-58a74b324bb7-kube-api-access-hw7gl\" (UniqueName: \"kubernetes.io/projected/ea61971c-5a72-4609-b5a0-58a74b324bb7-kube-api-access-hw7gl\") pod \"ea61971c-5a72-4609-b5a0-58a74b324bb7\" (UID: \"ea61971c-5a72-4609-b5a0-58a74b324bb7\") " Nov 02 01:43:47.679468 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.677820 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea61971c-5a72-4609-b5a0-58a74b324bb7-tls-certs\" (UniqueName: \"kubernetes.io/secret/ea61971c-5a72-4609-b5a0-58a74b324bb7-tls-certs\") pod \"ea61971c-5a72-4609-b5a0-58a74b324bb7\" (UID: \"ea61971c-5a72-4609-b5a0-58a74b324bb7\") " Nov 02 01:43:47.679468 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.677880 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea61971c-5a72-4609-b5a0-58a74b324bb7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea61971c-5a72-4609-b5a0-58a74b324bb7-tmp-dir\") pod \"ea61971c-5a72-4609-b5a0-58a74b324bb7\" (UID: \"ea61971c-5a72-4609-b5a0-58a74b324bb7\") " Nov 02 01:43:47.680673 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.680335 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea61971c-5a72-4609-b5a0-58a74b324bb7-home" pod "ea61971c-5a72-4609-b5a0-58a74b324bb7" (UID: "ea61971c-5a72-4609-b5a0-58a74b324bb7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:47.681752 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.680941 4427 scope.go:122] "RemoveContainer" containerID="5ad14d31ea4019e07ab69f7baf7c92eeb8d8e042e2f0fd5cf41634783065770a" Nov 02 01:43:47.681752 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.681245 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6-model-cache" pod "c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6" (UID: "c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:47.682605 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:43:47.682011 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ad14d31ea4019e07ab69f7baf7c92eeb8d8e042e2f0fd5cf41634783065770a\": container with ID starting with 5ad14d31ea4019e07ab69f7baf7c92eeb8d8e042e2f0fd5cf41634783065770a not found: ID does not exist" containerID="5ad14d31ea4019e07ab69f7baf7c92eeb8d8e042e2f0fd5cf41634783065770a" Nov 02 01:43:47.682605 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.682046 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ad14d31ea4019e07ab69f7baf7c92eeb8d8e042e2f0fd5cf41634783065770a"} err="failed to get container status \"5ad14d31ea4019e07ab69f7baf7c92eeb8d8e042e2f0fd5cf41634783065770a\": rpc error: code = NotFound desc = could not find container \"5ad14d31ea4019e07ab69f7baf7c92eeb8d8e042e2f0fd5cf41634783065770a\": container with ID starting with 5ad14d31ea4019e07ab69f7baf7c92eeb8d8e042e2f0fd5cf41634783065770a not found: ID does not exist" Nov 02 01:43:47.682605 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.682291 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea61971c-5a72-4609-b5a0-58a74b324bb7-tmp-dir" pod "ea61971c-5a72-4609-b5a0-58a74b324bb7" (UID: "ea61971c-5a72-4609-b5a0-58a74b324bb7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:47.684089 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.683826 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6-tmp-dir" pod "c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6" (UID: "c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:47.684689 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.684426 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6-home" pod "c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6" (UID: "c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:47.685008 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.684977 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea61971c-5a72-4609-b5a0-58a74b324bb7-model-cache" pod "ea61971c-5a72-4609-b5a0-58a74b324bb7" (UID: "ea61971c-5a72-4609-b5a0-58a74b324bb7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:47.694071 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.694043 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea61971c-5a72-4609-b5a0-58a74b324bb7-kube-api-access-hw7gl" pod "ea61971c-5a72-4609-b5a0-58a74b324bb7" (UID: "ea61971c-5a72-4609-b5a0-58a74b324bb7"). InnerVolumeSpecName "kube-api-access-hw7gl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:43:47.698249 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.698202 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6-dshm" pod "c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6" (UID: "c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:47.703973 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.703944 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea61971c-5a72-4609-b5a0-58a74b324bb7-dshm" pod "ea61971c-5a72-4609-b5a0-58a74b324bb7" (UID: "ea61971c-5a72-4609-b5a0-58a74b324bb7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:47.704357 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.704333 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea61971c-5a72-4609-b5a0-58a74b324bb7-tls-certs" pod "ea61971c-5a72-4609-b5a0-58a74b324bb7" (UID: "ea61971c-5a72-4609-b5a0-58a74b324bb7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:43:47.704846 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.704820 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6-kube-api-access-vz4fv" pod "c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6" (UID: "c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6"). InnerVolumeSpecName "kube-api-access-vz4fv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:43:47.705717 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.705692 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6-tls-certs" pod "c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6" (UID: "c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:43:47.735322 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.734863 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b34fc0-5b786c7bdc-8lrhc" Nov 02 01:43:47.794410 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.789337 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d10ccd31-3d14-498e-bf7a-0b3034576528-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d10ccd31-3d14-498e-bf7a-0b3034576528-plugins-config-volume\") pod \"d10ccd31-3d14-498e-bf7a-0b3034576528\" (UID: \"d10ccd31-3d14-498e-bf7a-0b3034576528\") " Nov 02 01:43:47.794410 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.789902 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d10ccd31-3d14-498e-bf7a-0b3034576528-kube-api-access-rp254\" (UniqueName: \"kubernetes.io/projected/d10ccd31-3d14-498e-bf7a-0b3034576528-kube-api-access-rp254\") pod \"d10ccd31-3d14-498e-bf7a-0b3034576528\" (UID: \"d10ccd31-3d14-498e-bf7a-0b3034576528\") " Nov 02 01:43:47.794410 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.790956 4427 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6-tls-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:47.794410 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.790982 4427 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6-dshm\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:47.794410 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.791008 4427 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6-tmp-dir\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:47.794410 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.791022 4427 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea61971c-5a72-4609-b5a0-58a74b324bb7-model-cache\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:47.794410 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.791040 4427 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea61971c-5a72-4609-b5a0-58a74b324bb7-dshm\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:47.794410 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.791056 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz4fv\" (UniqueName: \"kubernetes.io/projected/c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6-kube-api-access-vz4fv\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:47.794410 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.791070 4427 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6-home\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:47.794410 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.791083 4427 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6-model-cache\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:47.794410 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.791096 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hw7gl\" (UniqueName: \"kubernetes.io/projected/ea61971c-5a72-4609-b5a0-58a74b324bb7-kube-api-access-hw7gl\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:47.794410 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.791110 4427 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea61971c-5a72-4609-b5a0-58a74b324bb7-tls-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:47.794410 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.791123 4427 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea61971c-5a72-4609-b5a0-58a74b324bb7-tmp-dir\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:47.794410 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.791135 4427 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea61971c-5a72-4609-b5a0-58a74b324bb7-home\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:47.845402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.819524 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d10ccd31-3d14-498e-bf7a-0b3034576528-plugins-config-volume" pod "d10ccd31-3d14-498e-bf7a-0b3034576528" (UID: "d10ccd31-3d14-498e-bf7a-0b3034576528"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:43:47.889866 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.854615 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d10ccd31-3d14-498e-bf7a-0b3034576528-kube-api-access-rp254" pod "d10ccd31-3d14-498e-bf7a-0b3034576528" (UID: "d10ccd31-3d14-498e-bf7a-0b3034576528"). InnerVolumeSpecName "kube-api-access-rp254". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:43:47.987832 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.967712 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp254\" (UniqueName: \"kubernetes.io/projected/d10ccd31-3d14-498e-bf7a-0b3034576528-kube-api-access-rp254\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:47.987832 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:47.967748 4427 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d10ccd31-3d14-498e-bf7a-0b3034576528-plugins-config-volume\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:48.008955 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:48.008559 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b34fc0-6cd9f9c5fc-9n8zq" Nov 02 01:43:48.213515 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:48.213478 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs"] Nov 02 01:43:48.220854 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:48.220819 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs"] Nov 02 01:43:48.258429 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:48.254615 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e58e874-db28-4a0e-8b46-368b6d4d0960-kube-api-access-nghgl\" (UniqueName: \"kubernetes.io/projected/1e58e874-db28-4a0e-8b46-368b6d4d0960-kube-api-access-nghgl\") pod \"1e58e874-db28-4a0e-8b46-368b6d4d0960\" (UID: \"1e58e874-db28-4a0e-8b46-368b6d4d0960\") " Nov 02 01:43:48.275679 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:48.275623 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e58e874-db28-4a0e-8b46-368b6d4d0960-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e58e874-db28-4a0e-8b46-368b6d4d0960-plugins-config-volume\") pod \"1e58e874-db28-4a0e-8b46-368b6d4d0960\" (UID: \"1e58e874-db28-4a0e-8b46-368b6d4d0960\") " Nov 02 01:43:48.280498 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:48.280439 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e58e874-db28-4a0e-8b46-368b6d4d0960-plugins-config-volume" pod "1e58e874-db28-4a0e-8b46-368b6d4d0960" (UID: "1e58e874-db28-4a0e-8b46-368b6d4d0960"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:43:48.280926 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:48.280866 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b"] Nov 02 01:43:48.292345 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:48.292267 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-47dd19/distinct-w2-47dd19-kserve-95d95b88d-5m78b"] Nov 02 01:43:48.293484 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:48.293449 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e58e874-db28-4a0e-8b46-368b6d4d0960-kube-api-access-nghgl" pod "1e58e874-db28-4a0e-8b46-368b6d4d0960" (UID: "1e58e874-db28-4a0e-8b46-368b6d4d0960"). InnerVolumeSpecName "kube-api-access-nghgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:43:48.360196 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:48.360148 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47dd19/premium-w2-47dd19-kserve-68464dff57-wcgxs" podUID="c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:43:48.381912 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:48.381874 4427 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e58e874-db28-4a0e-8b46-368b6d4d0960-plugins-config-volume\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:48.381912 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:48.381909 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nghgl\" (UniqueName: \"kubernetes.io/projected/1e58e874-db28-4a0e-8b46-368b6d4d0960-kube-api-access-nghgl\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:48.468760 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:48.468722 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ffe5c1-579789c787-hz68t"] Nov 02 01:43:48.594037 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:48.593933 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b34fc0-5b786c7bdc-8lrhc" event={"ID":"d10ccd31-3d14-498e-bf7a-0b3034576528","Type":"ContainerDied","Data":"46b4e86e34bc0a6a6c0df5509b62f3cc63e2cd42351bfb89862d73618fe2a340"} Nov 02 01:43:48.594037 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:48.594017 4427 scope.go:122] "RemoveContainer" containerID="ba8d5bfd013544907336604267b6db353fb410700c119045e616171898fb616a" Nov 02 01:43:48.594319 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:48.593954 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b34fc0-5b786c7bdc-8lrhc" Nov 02 01:43:48.596532 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:48.596283 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b34fc0-6cd9f9c5fc-9n8zq" event={"ID":"1e58e874-db28-4a0e-8b46-368b6d4d0960","Type":"ContainerDied","Data":"1ee3dea6889ca41a43a16ee6c0e42d09daac35055dc0e032dab9d5392018c6a5"} Nov 02 01:43:48.596532 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:48.596449 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b34fc0-6cd9f9c5fc-9n8zq" Nov 02 01:43:48.619560 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:48.619524 4427 scope.go:122] "RemoveContainer" containerID="d65fa69dbf28c35abecfbb9bcfc5765dc93643496377c7228266dd0edfc22ec0" Nov 02 01:43:48.640449 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:48.640362 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b34fc0-6cd9f9c5fc-9n8zq"] Nov 02 01:43:48.646104 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:48.646076 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b34fc0-6cd9f9c5fc-9n8zq"] Nov 02 01:43:48.657041 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:48.657013 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b34fc0-5b786c7bdc-8lrhc"] Nov 02 01:43:48.669092 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:48.669062 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b34fc0-5b786c7bdc-8lrhc"] Nov 02 01:43:49.537265 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:49.537231 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e58e874-db28-4a0e-8b46-368b6d4d0960" path="/var/lib/kubelet/pods/1e58e874-db28-4a0e-8b46-368b6d4d0960/volumes" Nov 02 01:43:49.537845 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:49.537828 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6" path="/var/lib/kubelet/pods/c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6/volumes" Nov 02 01:43:49.538287 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:49.538273 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d10ccd31-3d14-498e-bf7a-0b3034576528" path="/var/lib/kubelet/pods/d10ccd31-3d14-498e-bf7a-0b3034576528/volumes" Nov 02 01:43:49.539101 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:49.539085 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea61971c-5a72-4609-b5a0-58a74b324bb7" path="/var/lib/kubelet/pods/ea61971c-5a72-4609-b5a0-58a74b324bb7/volumes" Nov 02 01:43:49.905962 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:49.905012 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2589b2-5c75d66b9-fkj6b"] Nov 02 01:43:49.905962 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:49.905686 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2589b2-5c75d66b9-fkj6b" podUID="3869bb0d-5d24-4a09-bd83-9aea716bfd90" containerName="maas-api" containerID="cri-o://e3bd476b484a40fd18b7ab169aebb10519356d01340b07ed4ce3f841ee22ef80" gracePeriod=30 Nov 02 01:43:50.362895 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:50.362067 4427 patch_prober.go:29] interesting pod/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 01:43:50.362895 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:50.362142 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4" podUID="c3d179f1-c7e0-4340-a286-2efbbc44ed53" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 01:43:50.624859 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:50.623912 4427 generic.go:356] "Generic (PLEG): container finished" podID="3869bb0d-5d24-4a09-bd83-9aea716bfd90" containerID="e3bd476b484a40fd18b7ab169aebb10519356d01340b07ed4ce3f841ee22ef80" exitCode=0 Nov 02 01:43:50.627526 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:50.626629 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2589b2-5c75d66b9-fkj6b" event={"ID":"3869bb0d-5d24-4a09-bd83-9aea716bfd90","Type":"ContainerDied","Data":"e3bd476b484a40fd18b7ab169aebb10519356d01340b07ed4ce3f841ee22ef80"} Nov 02 01:43:50.684035 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:50.684008 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2589b2-5c75d66b9-fkj6b" Nov 02 01:43:50.823669 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:50.823477 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3869bb0d-5d24-4a09-bd83-9aea716bfd90-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3869bb0d-5d24-4a09-bd83-9aea716bfd90-openshift-service-ca\") pod \"3869bb0d-5d24-4a09-bd83-9aea716bfd90\" (UID: \"3869bb0d-5d24-4a09-bd83-9aea716bfd90\") " Nov 02 01:43:50.824654 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:50.824490 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3869bb0d-5d24-4a09-bd83-9aea716bfd90-openshift-service-ca" pod "3869bb0d-5d24-4a09-bd83-9aea716bfd90" (UID: "3869bb0d-5d24-4a09-bd83-9aea716bfd90"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:43:50.825447 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:50.825425 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3869bb0d-5d24-4a09-bd83-9aea716bfd90-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3869bb0d-5d24-4a09-bd83-9aea716bfd90-maas-api-tls\") pod \"3869bb0d-5d24-4a09-bd83-9aea716bfd90\" (UID: \"3869bb0d-5d24-4a09-bd83-9aea716bfd90\") " Nov 02 01:43:50.826155 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:50.826131 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3869bb0d-5d24-4a09-bd83-9aea716bfd90-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3869bb0d-5d24-4a09-bd83-9aea716bfd90-metrics-certs\") pod \"3869bb0d-5d24-4a09-bd83-9aea716bfd90\" (UID: \"3869bb0d-5d24-4a09-bd83-9aea716bfd90\") " Nov 02 01:43:50.826857 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:50.826836 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3869bb0d-5d24-4a09-bd83-9aea716bfd90-kube-api-access-xrlxk\" (UniqueName: \"kubernetes.io/projected/3869bb0d-5d24-4a09-bd83-9aea716bfd90-kube-api-access-xrlxk\") pod \"3869bb0d-5d24-4a09-bd83-9aea716bfd90\" (UID: \"3869bb0d-5d24-4a09-bd83-9aea716bfd90\") " Nov 02 01:43:50.833155 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:50.833125 4427 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3869bb0d-5d24-4a09-bd83-9aea716bfd90-openshift-service-ca\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:50.840910 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:50.840867 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3869bb0d-5d24-4a09-bd83-9aea716bfd90-maas-api-tls" pod "3869bb0d-5d24-4a09-bd83-9aea716bfd90" (UID: "3869bb0d-5d24-4a09-bd83-9aea716bfd90"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:43:50.841056 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:50.840924 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3869bb0d-5d24-4a09-bd83-9aea716bfd90-metrics-certs" pod "3869bb0d-5d24-4a09-bd83-9aea716bfd90" (UID: "3869bb0d-5d24-4a09-bd83-9aea716bfd90"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:43:50.841056 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:50.840933 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3869bb0d-5d24-4a09-bd83-9aea716bfd90-kube-api-access-xrlxk" pod "3869bb0d-5d24-4a09-bd83-9aea716bfd90" (UID: "3869bb0d-5d24-4a09-bd83-9aea716bfd90"). InnerVolumeSpecName "kube-api-access-xrlxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:43:50.934513 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:50.934428 4427 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3869bb0d-5d24-4a09-bd83-9aea716bfd90-maas-api-tls\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:50.934513 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:50.934461 4427 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3869bb0d-5d24-4a09-bd83-9aea716bfd90-metrics-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:50.934513 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:50.934473 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrlxk\" (UniqueName: \"kubernetes.io/projected/3869bb0d-5d24-4a09-bd83-9aea716bfd90-kube-api-access-xrlxk\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:51.027050 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:51.027010 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2589b2-5b4dc6cc9f-nmtkc" podUID="a097a1f1-7f03-479d-a65a-90f8e542d06b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 01:43:51.637795 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:51.637765 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2589b2-5c75d66b9-fkj6b" Nov 02 01:43:51.637795 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:51.637785 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2589b2-5c75d66b9-fkj6b" event={"ID":"3869bb0d-5d24-4a09-bd83-9aea716bfd90","Type":"ContainerDied","Data":"094eb307744220bc21054a531a28e6577d06561c8f2410c254844d2e306e1f4f"} Nov 02 01:43:51.638331 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:51.637843 4427 scope.go:122] "RemoveContainer" containerID="e3bd476b484a40fd18b7ab169aebb10519356d01340b07ed4ce3f841ee22ef80" Nov 02 01:43:51.663612 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:51.663581 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2589b2-5c75d66b9-fkj6b"] Nov 02 01:43:51.668187 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:51.668165 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2589b2-5c75d66b9-fkj6b"] Nov 02 01:43:53.030431 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:53.030406 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4" Nov 02 01:43:53.178251 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:53.178213 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3d179f1-c7e0-4340-a286-2efbbc44ed53-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3d179f1-c7e0-4340-a286-2efbbc44ed53-istio-envoy\") pod \"c3d179f1-c7e0-4340-a286-2efbbc44ed53\" (UID: \"c3d179f1-c7e0-4340-a286-2efbbc44ed53\") " Nov 02 01:43:53.178421 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:53.178257 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c3d179f1-c7e0-4340-a286-2efbbc44ed53-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3d179f1-c7e0-4340-a286-2efbbc44ed53-istio-podinfo\") pod \"c3d179f1-c7e0-4340-a286-2efbbc44ed53\" (UID: \"c3d179f1-c7e0-4340-a286-2efbbc44ed53\") " Nov 02 01:43:53.178421 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:53.178345 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3d179f1-c7e0-4340-a286-2efbbc44ed53-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3d179f1-c7e0-4340-a286-2efbbc44ed53-istiod-ca-cert\") pod \"c3d179f1-c7e0-4340-a286-2efbbc44ed53\" (UID: \"c3d179f1-c7e0-4340-a286-2efbbc44ed53\") " Nov 02 01:43:53.178549 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:53.178471 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3d179f1-c7e0-4340-a286-2efbbc44ed53-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3d179f1-c7e0-4340-a286-2efbbc44ed53-credential-socket\") pod \"c3d179f1-c7e0-4340-a286-2efbbc44ed53\" (UID: \"c3d179f1-c7e0-4340-a286-2efbbc44ed53\") " Nov 02 01:43:53.178636 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:53.178599 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3d179f1-c7e0-4340-a286-2efbbc44ed53-kube-api-access-sl9pj\" (UniqueName: \"kubernetes.io/projected/c3d179f1-c7e0-4340-a286-2efbbc44ed53-kube-api-access-sl9pj\") pod \"c3d179f1-c7e0-4340-a286-2efbbc44ed53\" (UID: \"c3d179f1-c7e0-4340-a286-2efbbc44ed53\") " Nov 02 01:43:53.178706 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:53.178668 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3d179f1-c7e0-4340-a286-2efbbc44ed53-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3d179f1-c7e0-4340-a286-2efbbc44ed53-workload-certs\") pod \"c3d179f1-c7e0-4340-a286-2efbbc44ed53\" (UID: \"c3d179f1-c7e0-4340-a286-2efbbc44ed53\") " Nov 02 01:43:53.178806 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:53.178779 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3d179f1-c7e0-4340-a286-2efbbc44ed53-credential-socket" pod "c3d179f1-c7e0-4340-a286-2efbbc44ed53" (UID: "c3d179f1-c7e0-4340-a286-2efbbc44ed53"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:53.178872 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:53.178852 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3d179f1-c7e0-4340-a286-2efbbc44ed53-istiod-ca-cert" pod "c3d179f1-c7e0-4340-a286-2efbbc44ed53" (UID: "c3d179f1-c7e0-4340-a286-2efbbc44ed53"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:43:53.178977 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:53.178957 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3d179f1-c7e0-4340-a286-2efbbc44ed53-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3d179f1-c7e0-4340-a286-2efbbc44ed53-workload-socket\") pod \"c3d179f1-c7e0-4340-a286-2efbbc44ed53\" (UID: \"c3d179f1-c7e0-4340-a286-2efbbc44ed53\") " Nov 02 01:43:53.179053 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:53.179038 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3d179f1-c7e0-4340-a286-2efbbc44ed53-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3d179f1-c7e0-4340-a286-2efbbc44ed53-istio-data\") pod \"c3d179f1-c7e0-4340-a286-2efbbc44ed53\" (UID: \"c3d179f1-c7e0-4340-a286-2efbbc44ed53\") " Nov 02 01:43:53.179111 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:53.179053 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3d179f1-c7e0-4340-a286-2efbbc44ed53-workload-certs" pod "c3d179f1-c7e0-4340-a286-2efbbc44ed53" (UID: "c3d179f1-c7e0-4340-a286-2efbbc44ed53"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:53.179111 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:53.179093 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3d179f1-c7e0-4340-a286-2efbbc44ed53-istio-token\" (UniqueName: \"kubernetes.io/projected/c3d179f1-c7e0-4340-a286-2efbbc44ed53-istio-token\") pod \"c3d179f1-c7e0-4340-a286-2efbbc44ed53\" (UID: \"c3d179f1-c7e0-4340-a286-2efbbc44ed53\") " Nov 02 01:43:53.179277 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:53.179212 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3d179f1-c7e0-4340-a286-2efbbc44ed53-workload-socket" pod "c3d179f1-c7e0-4340-a286-2efbbc44ed53" (UID: "c3d179f1-c7e0-4340-a286-2efbbc44ed53"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:53.179450 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:53.179359 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3d179f1-c7e0-4340-a286-2efbbc44ed53-istio-data" pod "c3d179f1-c7e0-4340-a286-2efbbc44ed53" (UID: "c3d179f1-c7e0-4340-a286-2efbbc44ed53"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:53.180160 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:53.180132 4427 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3d179f1-c7e0-4340-a286-2efbbc44ed53-workload-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:53.180276 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:53.180163 4427 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3d179f1-c7e0-4340-a286-2efbbc44ed53-istio-data\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:53.180276 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:53.180177 4427 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3d179f1-c7e0-4340-a286-2efbbc44ed53-istiod-ca-cert\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:53.180276 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:53.180193 4427 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3d179f1-c7e0-4340-a286-2efbbc44ed53-credential-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:53.180276 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:53.180207 4427 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3d179f1-c7e0-4340-a286-2efbbc44ed53-workload-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:53.181150 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:53.181123 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3d179f1-c7e0-4340-a286-2efbbc44ed53-istio-envoy" pod "c3d179f1-c7e0-4340-a286-2efbbc44ed53" (UID: "c3d179f1-c7e0-4340-a286-2efbbc44ed53"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:53.181304 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:53.181286 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3d179f1-c7e0-4340-a286-2efbbc44ed53-istio-podinfo" pod "c3d179f1-c7e0-4340-a286-2efbbc44ed53" (UID: "c3d179f1-c7e0-4340-a286-2efbbc44ed53"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:43:53.181354 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:53.181299 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3d179f1-c7e0-4340-a286-2efbbc44ed53-kube-api-access-sl9pj" pod "c3d179f1-c7e0-4340-a286-2efbbc44ed53" (UID: "c3d179f1-c7e0-4340-a286-2efbbc44ed53"). InnerVolumeSpecName "kube-api-access-sl9pj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:43:53.181613 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:53.181592 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3d179f1-c7e0-4340-a286-2efbbc44ed53-istio-token" pod "c3d179f1-c7e0-4340-a286-2efbbc44ed53" (UID: "c3d179f1-c7e0-4340-a286-2efbbc44ed53"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:43:53.282659 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:53.282563 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sl9pj\" (UniqueName: \"kubernetes.io/projected/c3d179f1-c7e0-4340-a286-2efbbc44ed53-kube-api-access-sl9pj\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:53.282659 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:53.282596 4427 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3d179f1-c7e0-4340-a286-2efbbc44ed53-istio-token\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:53.282659 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:53.282610 4427 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3d179f1-c7e0-4340-a286-2efbbc44ed53-istio-envoy\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:53.282659 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:53.282622 4427 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3d179f1-c7e0-4340-a286-2efbbc44ed53-istio-podinfo\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:43:53.469898 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:53.469848 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-ffe5c1-579789c787-hz68t" podUID="b9bb7aa5-03de-42b9-a6d3-b1d8e7806a74" containerName="payload-processing" containerID="cri-o://de6f1c505982666c93a25403b9eea087c169cd8dac0531c888ef4a3351663a60" gracePeriod=25 Nov 02 01:43:53.536987 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:53.536897 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3869bb0d-5d24-4a09-bd83-9aea716bfd90" path="/var/lib/kubelet/pods/3869bb0d-5d24-4a09-bd83-9aea716bfd90/volumes" Nov 02 01:43:53.659040 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:53.659008 4427 generic.go:356] "Generic (PLEG): container finished" podID="c3d179f1-c7e0-4340-a286-2efbbc44ed53" containerID="a7ff2e236648abc871bf55a084d15da6ae70b7df221bc21f7fa2e80d0df88c64" exitCode=0 Nov 02 01:43:53.659213 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:53.659106 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4" Nov 02 01:43:53.659213 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:53.659137 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4" event={"ID":"c3d179f1-c7e0-4340-a286-2efbbc44ed53","Type":"ContainerDied","Data":"a7ff2e236648abc871bf55a084d15da6ae70b7df221bc21f7fa2e80d0df88c64"} Nov 02 01:43:53.659213 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:53.659191 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4" event={"ID":"c3d179f1-c7e0-4340-a286-2efbbc44ed53","Type":"ContainerDied","Data":"88900bf442a24ca8fdbb3b9c414e93bb9a8e230cc86eabf5c29c885c6e904ae2"} Nov 02 01:43:53.659213 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:53.659213 4427 scope.go:122] "RemoveContainer" containerID="a7ff2e236648abc871bf55a084d15da6ae70b7df221bc21f7fa2e80d0df88c64" Nov 02 01:43:53.669870 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:53.669838 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-ffe5c1-579789c787-hz68t" podUID="b9bb7aa5-03de-42b9-a6d3-b1d8e7806a74" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 01:43:53.670761 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:53.670739 4427 scope.go:122] "RemoveContainer" containerID="662874da8f50b0842d9938672194dc31d4755f470d32aaa4db1518c240e7bfa5" Nov 02 01:43:53.684335 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:53.684307 4427 scope.go:122] "RemoveContainer" containerID="a7ff2e236648abc871bf55a084d15da6ae70b7df221bc21f7fa2e80d0df88c64" Nov 02 01:43:53.684752 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:43:53.684721 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7ff2e236648abc871bf55a084d15da6ae70b7df221bc21f7fa2e80d0df88c64\": container with ID starting with a7ff2e236648abc871bf55a084d15da6ae70b7df221bc21f7fa2e80d0df88c64 not found: ID does not exist" containerID="a7ff2e236648abc871bf55a084d15da6ae70b7df221bc21f7fa2e80d0df88c64" Nov 02 01:43:53.684841 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:53.684767 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7ff2e236648abc871bf55a084d15da6ae70b7df221bc21f7fa2e80d0df88c64"} err="failed to get container status \"a7ff2e236648abc871bf55a084d15da6ae70b7df221bc21f7fa2e80d0df88c64\": rpc error: code = NotFound desc = could not find container \"a7ff2e236648abc871bf55a084d15da6ae70b7df221bc21f7fa2e80d0df88c64\": container with ID starting with a7ff2e236648abc871bf55a084d15da6ae70b7df221bc21f7fa2e80d0df88c64 not found: ID does not exist" Nov 02 01:43:53.684841 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:53.684794 4427 scope.go:122] "RemoveContainer" containerID="662874da8f50b0842d9938672194dc31d4755f470d32aaa4db1518c240e7bfa5" Nov 02 01:43:53.684841 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:53.684826 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4"] Nov 02 01:43:53.685168 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:43:53.685146 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"662874da8f50b0842d9938672194dc31d4755f470d32aaa4db1518c240e7bfa5\": container with ID starting with 662874da8f50b0842d9938672194dc31d4755f470d32aaa4db1518c240e7bfa5 not found: ID does not exist" containerID="662874da8f50b0842d9938672194dc31d4755f470d32aaa4db1518c240e7bfa5" Nov 02 01:43:53.685227 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:53.685179 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"662874da8f50b0842d9938672194dc31d4755f470d32aaa4db1518c240e7bfa5"} err="failed to get container status \"662874da8f50b0842d9938672194dc31d4755f470d32aaa4db1518c240e7bfa5\": rpc error: code = NotFound desc = could not find container \"662874da8f50b0842d9938672194dc31d4755f470d32aaa4db1518c240e7bfa5\": container with ID starting with 662874da8f50b0842d9938672194dc31d4755f470d32aaa4db1518c240e7bfa5 not found: ID does not exist" Nov 02 01:43:53.689571 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:53.689551 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-b34fc0-openshift-default-7d8f65d947-rrpn4"] Nov 02 01:43:55.564461 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:55.564422 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3d179f1-c7e0-4340-a286-2efbbc44ed53" path="/var/lib/kubelet/pods/c3d179f1-c7e0-4340-a286-2efbbc44ed53/volumes" Nov 02 01:43:55.564988 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:55.564928 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x"] Nov 02 01:43:55.565272 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:55.565173 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x" podUID="e1412019-62bb-4fc3-a7ad-8e123fef90eb" containerName="istio-proxy" containerID="cri-o://66f3e69d92a5ab91170fb508ebaa7f30900ee72f6a5a4e3fce4255db6ddb5fc6" gracePeriod=30 Nov 02 01:43:56.469827 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:56.469786 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-c22a0d-569bff79c5-8qj76"] Nov 02 01:43:56.483735 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:56.483704 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c22a0d-7bd45b488c-gr9gq"] Nov 02 01:43:58.426577 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:58.426535 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g"] Nov 02 01:43:58.427071 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:58.426800 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g" podUID="6b8221ff-cb53-42c0-a697-4992c40d8e52" containerName="istio-proxy" containerID="cri-o://1758ac1393ccd3d867b999af3e71362b1a13d3afb4e9c7a6a6da7c8c16705754" gracePeriod=30 Nov 02 01:43:58.529466 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:58.528949 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9a4595-5d57f7c66f-5vgsh"] Nov 02 01:43:58.536536 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:58.536500 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9a4595-cc768d49f-fb6l8"] Nov 02 01:43:59.032757 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:59.030723 4427 patch_prober.go:29] interesting pod/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 01:43:59.032757 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:43:59.031114 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x" podUID="e1412019-62bb-4fc3-a7ad-8e123fef90eb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 01:44:00.810203 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:00.810161 4427 generic.go:356] "Generic (PLEG): container finished" podID="e1412019-62bb-4fc3-a7ad-8e123fef90eb" containerID="66f3e69d92a5ab91170fb508ebaa7f30900ee72f6a5a4e3fce4255db6ddb5fc6" exitCode=0 Nov 02 01:44:00.810718 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:00.810280 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x" event={"ID":"e1412019-62bb-4fc3-a7ad-8e123fef90eb","Type":"ContainerDied","Data":"66f3e69d92a5ab91170fb508ebaa7f30900ee72f6a5a4e3fce4255db6ddb5fc6"} Nov 02 01:44:00.908189 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:00.908159 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x" Nov 02 01:44:01.024055 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:01.023666 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9a4595-554977c4cb-vhq5g"] Nov 02 01:44:01.024055 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:01.023919 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a4595-554977c4cb-vhq5g" podUID="57445e45-2af1-475b-a32b-e8d88a5cdc54" containerName="maas-api" containerID="cri-o://d7affb1576cabd70041a3aae992bad6e04a89aa969e6749d094677fa63ac66d2" gracePeriod=30 Nov 02 01:44:01.026984 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:01.026946 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2589b2-5b4dc6cc9f-nmtkc" podUID="a097a1f1-7f03-479d-a65a-90f8e542d06b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 01:44:01.064281 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:01.063509 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1412019-62bb-4fc3-a7ad-8e123fef90eb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1412019-62bb-4fc3-a7ad-8e123fef90eb-istio-envoy\") pod \"e1412019-62bb-4fc3-a7ad-8e123fef90eb\" (UID: \"e1412019-62bb-4fc3-a7ad-8e123fef90eb\") " Nov 02 01:44:01.064281 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:01.063619 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1412019-62bb-4fc3-a7ad-8e123fef90eb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1412019-62bb-4fc3-a7ad-8e123fef90eb-workload-socket\") pod \"e1412019-62bb-4fc3-a7ad-8e123fef90eb\" (UID: \"e1412019-62bb-4fc3-a7ad-8e123fef90eb\") " Nov 02 01:44:01.064281 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:01.063656 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1412019-62bb-4fc3-a7ad-8e123fef90eb-kube-api-access-2dlff\" (UniqueName: \"kubernetes.io/projected/e1412019-62bb-4fc3-a7ad-8e123fef90eb-kube-api-access-2dlff\") pod \"e1412019-62bb-4fc3-a7ad-8e123fef90eb\" (UID: \"e1412019-62bb-4fc3-a7ad-8e123fef90eb\") " Nov 02 01:44:01.064281 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:01.063718 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e1412019-62bb-4fc3-a7ad-8e123fef90eb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1412019-62bb-4fc3-a7ad-8e123fef90eb-istio-podinfo\") pod \"e1412019-62bb-4fc3-a7ad-8e123fef90eb\" (UID: \"e1412019-62bb-4fc3-a7ad-8e123fef90eb\") " Nov 02 01:44:01.064281 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:01.063938 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1412019-62bb-4fc3-a7ad-8e123fef90eb-istio-token\" (UniqueName: \"kubernetes.io/projected/e1412019-62bb-4fc3-a7ad-8e123fef90eb-istio-token\") pod \"e1412019-62bb-4fc3-a7ad-8e123fef90eb\" (UID: \"e1412019-62bb-4fc3-a7ad-8e123fef90eb\") " Nov 02 01:44:01.064281 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:01.064007 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1412019-62bb-4fc3-a7ad-8e123fef90eb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1412019-62bb-4fc3-a7ad-8e123fef90eb-istiod-ca-cert\") pod \"e1412019-62bb-4fc3-a7ad-8e123fef90eb\" (UID: \"e1412019-62bb-4fc3-a7ad-8e123fef90eb\") " Nov 02 01:44:01.064281 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:01.064080 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1412019-62bb-4fc3-a7ad-8e123fef90eb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1412019-62bb-4fc3-a7ad-8e123fef90eb-workload-certs\") pod \"e1412019-62bb-4fc3-a7ad-8e123fef90eb\" (UID: \"e1412019-62bb-4fc3-a7ad-8e123fef90eb\") " Nov 02 01:44:01.064281 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:01.064164 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1412019-62bb-4fc3-a7ad-8e123fef90eb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1412019-62bb-4fc3-a7ad-8e123fef90eb-credential-socket\") pod \"e1412019-62bb-4fc3-a7ad-8e123fef90eb\" (UID: \"e1412019-62bb-4fc3-a7ad-8e123fef90eb\") " Nov 02 01:44:01.064281 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:01.064213 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1412019-62bb-4fc3-a7ad-8e123fef90eb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1412019-62bb-4fc3-a7ad-8e123fef90eb-istio-data\") pod \"e1412019-62bb-4fc3-a7ad-8e123fef90eb\" (UID: \"e1412019-62bb-4fc3-a7ad-8e123fef90eb\") " Nov 02 01:44:01.079890 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:01.078566 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1412019-62bb-4fc3-a7ad-8e123fef90eb-credential-socket" pod "e1412019-62bb-4fc3-a7ad-8e123fef90eb" (UID: "e1412019-62bb-4fc3-a7ad-8e123fef90eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:01.079890 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:01.078708 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1412019-62bb-4fc3-a7ad-8e123fef90eb-workload-certs" pod "e1412019-62bb-4fc3-a7ad-8e123fef90eb" (UID: "e1412019-62bb-4fc3-a7ad-8e123fef90eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:01.079890 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:01.079096 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1412019-62bb-4fc3-a7ad-8e123fef90eb-workload-socket" pod "e1412019-62bb-4fc3-a7ad-8e123fef90eb" (UID: "e1412019-62bb-4fc3-a7ad-8e123fef90eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:01.079890 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:01.079171 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1412019-62bb-4fc3-a7ad-8e123fef90eb-istio-data" pod "e1412019-62bb-4fc3-a7ad-8e123fef90eb" (UID: "e1412019-62bb-4fc3-a7ad-8e123fef90eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:01.079890 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:01.079851 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1412019-62bb-4fc3-a7ad-8e123fef90eb-istiod-ca-cert" pod "e1412019-62bb-4fc3-a7ad-8e123fef90eb" (UID: "e1412019-62bb-4fc3-a7ad-8e123fef90eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:44:01.092227 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:01.092191 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1412019-62bb-4fc3-a7ad-8e123fef90eb-kube-api-access-2dlff" pod "e1412019-62bb-4fc3-a7ad-8e123fef90eb" (UID: "e1412019-62bb-4fc3-a7ad-8e123fef90eb"). InnerVolumeSpecName "kube-api-access-2dlff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:44:01.092480 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:01.092454 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1412019-62bb-4fc3-a7ad-8e123fef90eb-istio-token" pod "e1412019-62bb-4fc3-a7ad-8e123fef90eb" (UID: "e1412019-62bb-4fc3-a7ad-8e123fef90eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:44:01.094461 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:01.094000 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1412019-62bb-4fc3-a7ad-8e123fef90eb-istio-envoy" pod "e1412019-62bb-4fc3-a7ad-8e123fef90eb" (UID: "e1412019-62bb-4fc3-a7ad-8e123fef90eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:01.094655 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:01.094626 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e1412019-62bb-4fc3-a7ad-8e123fef90eb-istio-podinfo" pod "e1412019-62bb-4fc3-a7ad-8e123fef90eb" (UID: "e1412019-62bb-4fc3-a7ad-8e123fef90eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:44:01.167063 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:01.166845 4427 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1412019-62bb-4fc3-a7ad-8e123fef90eb-credential-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:01.167063 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:01.166884 4427 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1412019-62bb-4fc3-a7ad-8e123fef90eb-istio-data\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:01.167063 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:01.166898 4427 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1412019-62bb-4fc3-a7ad-8e123fef90eb-istio-envoy\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:01.167063 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:01.166912 4427 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1412019-62bb-4fc3-a7ad-8e123fef90eb-workload-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:01.167063 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:01.166926 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dlff\" (UniqueName: \"kubernetes.io/projected/e1412019-62bb-4fc3-a7ad-8e123fef90eb-kube-api-access-2dlff\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:01.167063 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:01.166942 4427 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1412019-62bb-4fc3-a7ad-8e123fef90eb-istio-podinfo\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:01.167063 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:01.166957 4427 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1412019-62bb-4fc3-a7ad-8e123fef90eb-istio-token\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:01.167063 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:01.166970 4427 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1412019-62bb-4fc3-a7ad-8e123fef90eb-istiod-ca-cert\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:01.167063 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:01.166983 4427 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1412019-62bb-4fc3-a7ad-8e123fef90eb-workload-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:01.510415 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:01.487470 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c22a0d-7bd45b488c-gr9gq" podUID="0ed6780b-6074-4ce4-b9c0-44253b32955c" containerName="payload-pre-processing" containerID="cri-o://6f0c66960a98bd6033aa4054cfb10407e08744e16b2c93963d60e3bbc0d64da1" gracePeriod=25 Nov 02 01:44:01.510415 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:01.488064 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c22a0d-569bff79c5-8qj76" podUID="23707aeb-a006-4049-bca4-074bbd446ca7" containerName="payload-processing" containerID="cri-o://d7476636108306de61328f8070f68e0a4ad18b97a1e992c1c59e3289e3841d81" gracePeriod=25 Nov 02 01:44:01.583810 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:01.573755 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a4595-554977c4cb-vhq5g" Nov 02 01:44:01.703457 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:01.690393 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57445e45-2af1-475b-a32b-e8d88a5cdc54-kube-api-access-5wzpw\" (UniqueName: \"kubernetes.io/projected/57445e45-2af1-475b-a32b-e8d88a5cdc54-kube-api-access-5wzpw\") pod \"57445e45-2af1-475b-a32b-e8d88a5cdc54\" (UID: \"57445e45-2af1-475b-a32b-e8d88a5cdc54\") " Nov 02 01:44:01.703457 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:01.690576 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57445e45-2af1-475b-a32b-e8d88a5cdc54-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57445e45-2af1-475b-a32b-e8d88a5cdc54-openshift-service-ca\") pod \"57445e45-2af1-475b-a32b-e8d88a5cdc54\" (UID: \"57445e45-2af1-475b-a32b-e8d88a5cdc54\") " Nov 02 01:44:01.703457 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:01.690662 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57445e45-2af1-475b-a32b-e8d88a5cdc54-metrics-certs\" (UniqueName: \"kubernetes.io/secret/57445e45-2af1-475b-a32b-e8d88a5cdc54-metrics-certs\") pod \"57445e45-2af1-475b-a32b-e8d88a5cdc54\" (UID: \"57445e45-2af1-475b-a32b-e8d88a5cdc54\") " Nov 02 01:44:01.703457 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:01.690891 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57445e45-2af1-475b-a32b-e8d88a5cdc54-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57445e45-2af1-475b-a32b-e8d88a5cdc54-maas-api-tls\") pod \"57445e45-2af1-475b-a32b-e8d88a5cdc54\" (UID: \"57445e45-2af1-475b-a32b-e8d88a5cdc54\") " Nov 02 01:44:01.727401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:01.718690 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57445e45-2af1-475b-a32b-e8d88a5cdc54-openshift-service-ca" pod "57445e45-2af1-475b-a32b-e8d88a5cdc54" (UID: "57445e45-2af1-475b-a32b-e8d88a5cdc54"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:44:01.858650 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:01.858612 4427 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57445e45-2af1-475b-a32b-e8d88a5cdc54-openshift-service-ca\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:01.918398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:01.908512 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57445e45-2af1-475b-a32b-e8d88a5cdc54-maas-api-tls" pod "57445e45-2af1-475b-a32b-e8d88a5cdc54" (UID: "57445e45-2af1-475b-a32b-e8d88a5cdc54"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:44:01.935493 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:01.935152 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57445e45-2af1-475b-a32b-e8d88a5cdc54-kube-api-access-5wzpw" pod "57445e45-2af1-475b-a32b-e8d88a5cdc54" (UID: "57445e45-2af1-475b-a32b-e8d88a5cdc54"). InnerVolumeSpecName "kube-api-access-5wzpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:44:01.949429 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:01.941863 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x" event={"ID":"e1412019-62bb-4fc3-a7ad-8e123fef90eb","Type":"ContainerDied","Data":"29708cb6ab707868d112cb40dae2225967bd7d66e70c6a04d5e048e42c366e9e"} Nov 02 01:44:01.949429 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:01.941939 4427 scope.go:122] "RemoveContainer" containerID="66f3e69d92a5ab91170fb508ebaa7f30900ee72f6a5a4e3fce4255db6ddb5fc6" Nov 02 01:44:01.949429 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:01.942091 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x" Nov 02 01:44:01.949429 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:01.947648 4427 generic.go:356] "Generic (PLEG): container finished" podID="57445e45-2af1-475b-a32b-e8d88a5cdc54" containerID="d7affb1576cabd70041a3aae992bad6e04a89aa969e6749d094677fa63ac66d2" exitCode=0 Nov 02 01:44:01.949429 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:01.947789 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a4595-554977c4cb-vhq5g" event={"ID":"57445e45-2af1-475b-a32b-e8d88a5cdc54","Type":"ContainerDied","Data":"d7affb1576cabd70041a3aae992bad6e04a89aa969e6749d094677fa63ac66d2"} Nov 02 01:44:01.949429 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:01.947841 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a4595-554977c4cb-vhq5g" event={"ID":"57445e45-2af1-475b-a32b-e8d88a5cdc54","Type":"ContainerDied","Data":"79190fe346b45022db532e3e229fa88cae4290a82762cae4be18470db486c2b9"} Nov 02 01:44:01.949429 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:01.947959 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a4595-554977c4cb-vhq5g" Nov 02 01:44:02.021346 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:01.978826 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57445e45-2af1-475b-a32b-e8d88a5cdc54-metrics-certs" pod "57445e45-2af1-475b-a32b-e8d88a5cdc54" (UID: "57445e45-2af1-475b-a32b-e8d88a5cdc54"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:44:02.021346 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:01.979338 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57445e45-2af1-475b-a32b-e8d88a5cdc54-metrics-certs\" (UniqueName: \"kubernetes.io/secret/57445e45-2af1-475b-a32b-e8d88a5cdc54-metrics-certs\") pod \"57445e45-2af1-475b-a32b-e8d88a5cdc54\" (UID: \"57445e45-2af1-475b-a32b-e8d88a5cdc54\") " Nov 02 01:44:02.021346 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:01.982141 4427 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57445e45-2af1-475b-a32b-e8d88a5cdc54-maas-api-tls\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:02.021346 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:01.982168 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wzpw\" (UniqueName: \"kubernetes.io/projected/57445e45-2af1-475b-a32b-e8d88a5cdc54-kube-api-access-5wzpw\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:02.021346 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:44:01.982255 4427 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/57445e45-2af1-475b-a32b-e8d88a5cdc54/volumes/kubernetes.io~secret/metrics-certs Nov 02 01:44:02.021346 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:01.982276 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57445e45-2af1-475b-a32b-e8d88a5cdc54-metrics-certs" pod "57445e45-2af1-475b-a32b-e8d88a5cdc54" (UID: "57445e45-2af1-475b-a32b-e8d88a5cdc54"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:44:02.021346 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:02.019006 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x"] Nov 02 01:44:02.021346 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:44:02.019314 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:44:02.050674 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:02.050636 4427 scope.go:122] "RemoveContainer" containerID="d7affb1576cabd70041a3aae992bad6e04a89aa969e6749d094677fa63ac66d2" Nov 02 01:44:02.059224 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:02.059187 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-wxn6x"] Nov 02 01:44:02.068586 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:02.068553 4427 scope.go:122] "RemoveContainer" containerID="d7affb1576cabd70041a3aae992bad6e04a89aa969e6749d094677fa63ac66d2" Nov 02 01:44:02.071572 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:44:02.071538 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7affb1576cabd70041a3aae992bad6e04a89aa969e6749d094677fa63ac66d2\": container with ID starting with d7affb1576cabd70041a3aae992bad6e04a89aa969e6749d094677fa63ac66d2 not found: ID does not exist" containerID="d7affb1576cabd70041a3aae992bad6e04a89aa969e6749d094677fa63ac66d2" Nov 02 01:44:02.071785 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:02.071756 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7affb1576cabd70041a3aae992bad6e04a89aa969e6749d094677fa63ac66d2"} err="failed to get container status \"d7affb1576cabd70041a3aae992bad6e04a89aa969e6749d094677fa63ac66d2\": rpc error: code = NotFound desc = could not find container \"d7affb1576cabd70041a3aae992bad6e04a89aa969e6749d094677fa63ac66d2\": container with ID starting with d7affb1576cabd70041a3aae992bad6e04a89aa969e6749d094677fa63ac66d2 not found: ID does not exist" Nov 02 01:44:02.087044 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:02.087009 4427 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57445e45-2af1-475b-a32b-e8d88a5cdc54-metrics-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:02.405498 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:02.403355 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9a4595-554977c4cb-vhq5g"] Nov 02 01:44:02.405498 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:02.404862 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c22a0d-569bff79c5-8qj76" Nov 02 01:44:02.409729 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:02.409701 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c22a0d-7bd45b488c-gr9gq" Nov 02 01:44:02.458828 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:02.458797 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9a4595-554977c4cb-vhq5g"] Nov 02 01:44:02.505999 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:02.505907 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23707aeb-a006-4049-bca4-074bbd446ca7-kube-api-access-l69dk\" (UniqueName: \"kubernetes.io/projected/23707aeb-a006-4049-bca4-074bbd446ca7-kube-api-access-l69dk\") pod \"23707aeb-a006-4049-bca4-074bbd446ca7\" (UID: \"23707aeb-a006-4049-bca4-074bbd446ca7\") " Nov 02 01:44:02.506179 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:02.506040 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ed6780b-6074-4ce4-b9c0-44253b32955c-kube-api-access-j8nbm\" (UniqueName: \"kubernetes.io/projected/0ed6780b-6074-4ce4-b9c0-44253b32955c-kube-api-access-j8nbm\") pod \"0ed6780b-6074-4ce4-b9c0-44253b32955c\" (UID: \"0ed6780b-6074-4ce4-b9c0-44253b32955c\") " Nov 02 01:44:02.506325 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:02.506302 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ed6780b-6074-4ce4-b9c0-44253b32955c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ed6780b-6074-4ce4-b9c0-44253b32955c-plugins-config-volume\") pod \"0ed6780b-6074-4ce4-b9c0-44253b32955c\" (UID: \"0ed6780b-6074-4ce4-b9c0-44253b32955c\") " Nov 02 01:44:02.506417 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:02.506360 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23707aeb-a006-4049-bca4-074bbd446ca7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23707aeb-a006-4049-bca4-074bbd446ca7-plugins-config-volume\") pod \"23707aeb-a006-4049-bca4-074bbd446ca7\" (UID: \"23707aeb-a006-4049-bca4-074bbd446ca7\") " Nov 02 01:44:02.506887 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:02.506855 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ed6780b-6074-4ce4-b9c0-44253b32955c-plugins-config-volume" pod "0ed6780b-6074-4ce4-b9c0-44253b32955c" (UID: "0ed6780b-6074-4ce4-b9c0-44253b32955c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:44:02.507281 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:02.507254 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23707aeb-a006-4049-bca4-074bbd446ca7-plugins-config-volume" pod "23707aeb-a006-4049-bca4-074bbd446ca7" (UID: "23707aeb-a006-4049-bca4-074bbd446ca7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:44:02.507545 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:02.507521 4427 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ed6780b-6074-4ce4-b9c0-44253b32955c-plugins-config-volume\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:02.507631 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:02.507552 4427 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23707aeb-a006-4049-bca4-074bbd446ca7-plugins-config-volume\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:02.510907 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:02.510852 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23707aeb-a006-4049-bca4-074bbd446ca7-kube-api-access-l69dk" pod "23707aeb-a006-4049-bca4-074bbd446ca7" (UID: "23707aeb-a006-4049-bca4-074bbd446ca7"). InnerVolumeSpecName "kube-api-access-l69dk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:44:02.512424 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:02.512396 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ed6780b-6074-4ce4-b9c0-44253b32955c-kube-api-access-j8nbm" pod "0ed6780b-6074-4ce4-b9c0-44253b32955c" (UID: "0ed6780b-6074-4ce4-b9c0-44253b32955c"). InnerVolumeSpecName "kube-api-access-j8nbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:44:02.608930 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:02.608891 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l69dk\" (UniqueName: \"kubernetes.io/projected/23707aeb-a006-4049-bca4-074bbd446ca7-kube-api-access-l69dk\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:02.608930 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:02.608928 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8nbm\" (UniqueName: \"kubernetes.io/projected/0ed6780b-6074-4ce4-b9c0-44253b32955c-kube-api-access-j8nbm\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:02.960220 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:02.960174 4427 generic.go:356] "Generic (PLEG): container finished" podID="0ed6780b-6074-4ce4-b9c0-44253b32955c" containerID="6f0c66960a98bd6033aa4054cfb10407e08744e16b2c93963d60e3bbc0d64da1" exitCode=0 Nov 02 01:44:02.960785 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:02.960227 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c22a0d-7bd45b488c-gr9gq" Nov 02 01:44:02.960785 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:02.960270 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c22a0d-7bd45b488c-gr9gq" event={"ID":"0ed6780b-6074-4ce4-b9c0-44253b32955c","Type":"ContainerDied","Data":"6f0c66960a98bd6033aa4054cfb10407e08744e16b2c93963d60e3bbc0d64da1"} Nov 02 01:44:02.960785 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:02.960334 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c22a0d-7bd45b488c-gr9gq" event={"ID":"0ed6780b-6074-4ce4-b9c0-44253b32955c","Type":"ContainerDied","Data":"650e767fd9c9231ea0b9605a4c571309b638dcee92b7761f9283c52fd0cceaec"} Nov 02 01:44:02.960785 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:02.960365 4427 scope.go:122] "RemoveContainer" containerID="6f0c66960a98bd6033aa4054cfb10407e08744e16b2c93963d60e3bbc0d64da1" Nov 02 01:44:02.962885 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:02.962736 4427 generic.go:356] "Generic (PLEG): container finished" podID="23707aeb-a006-4049-bca4-074bbd446ca7" containerID="d7476636108306de61328f8070f68e0a4ad18b97a1e992c1c59e3289e3841d81" exitCode=0 Nov 02 01:44:02.962885 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:02.962812 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c22a0d-569bff79c5-8qj76" Nov 02 01:44:02.962885 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:02.962874 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c22a0d-569bff79c5-8qj76" event={"ID":"23707aeb-a006-4049-bca4-074bbd446ca7","Type":"ContainerDied","Data":"d7476636108306de61328f8070f68e0a4ad18b97a1e992c1c59e3289e3841d81"} Nov 02 01:44:02.963128 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:02.962912 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c22a0d-569bff79c5-8qj76" event={"ID":"23707aeb-a006-4049-bca4-074bbd446ca7","Type":"ContainerDied","Data":"cd6f54a3f22e95c234511cbf0069f122b2f970f17db3d8621328ddec9bd532f6"} Nov 02 01:44:02.976847 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:02.976212 4427 scope.go:122] "RemoveContainer" containerID="6f0c66960a98bd6033aa4054cfb10407e08744e16b2c93963d60e3bbc0d64da1" Nov 02 01:44:02.976847 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:44:02.976682 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f0c66960a98bd6033aa4054cfb10407e08744e16b2c93963d60e3bbc0d64da1\": container with ID starting with 6f0c66960a98bd6033aa4054cfb10407e08744e16b2c93963d60e3bbc0d64da1 not found: ID does not exist" containerID="6f0c66960a98bd6033aa4054cfb10407e08744e16b2c93963d60e3bbc0d64da1" Nov 02 01:44:02.976847 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:02.976720 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f0c66960a98bd6033aa4054cfb10407e08744e16b2c93963d60e3bbc0d64da1"} err="failed to get container status \"6f0c66960a98bd6033aa4054cfb10407e08744e16b2c93963d60e3bbc0d64da1\": rpc error: code = NotFound desc = could not find container \"6f0c66960a98bd6033aa4054cfb10407e08744e16b2c93963d60e3bbc0d64da1\": container with ID starting with 6f0c66960a98bd6033aa4054cfb10407e08744e16b2c93963d60e3bbc0d64da1 not found: ID does not exist" Nov 02 01:44:02.976847 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:02.976745 4427 scope.go:122] "RemoveContainer" containerID="d7476636108306de61328f8070f68e0a4ad18b97a1e992c1c59e3289e3841d81" Nov 02 01:44:02.995702 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:02.995013 4427 scope.go:122] "RemoveContainer" containerID="d7476636108306de61328f8070f68e0a4ad18b97a1e992c1c59e3289e3841d81" Nov 02 01:44:02.995702 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:44:02.995427 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7476636108306de61328f8070f68e0a4ad18b97a1e992c1c59e3289e3841d81\": container with ID starting with d7476636108306de61328f8070f68e0a4ad18b97a1e992c1c59e3289e3841d81 not found: ID does not exist" containerID="d7476636108306de61328f8070f68e0a4ad18b97a1e992c1c59e3289e3841d81" Nov 02 01:44:02.995702 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:02.995456 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7476636108306de61328f8070f68e0a4ad18b97a1e992c1c59e3289e3841d81"} err="failed to get container status \"d7476636108306de61328f8070f68e0a4ad18b97a1e992c1c59e3289e3841d81\": rpc error: code = NotFound desc = could not find container \"d7476636108306de61328f8070f68e0a4ad18b97a1e992c1c59e3289e3841d81\": container with ID starting with d7476636108306de61328f8070f68e0a4ad18b97a1e992c1c59e3289e3841d81 not found: ID does not exist" Nov 02 01:44:03.042863 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:03.042824 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-c22a0d-569bff79c5-8qj76"] Nov 02 01:44:03.063150 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:03.063121 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-c22a0d-569bff79c5-8qj76"] Nov 02 01:44:03.077975 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:03.076643 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c22a0d-7bd45b488c-gr9gq"] Nov 02 01:44:03.095950 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:03.095913 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c22a0d-7bd45b488c-gr9gq"] Nov 02 01:44:03.547915 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:03.546807 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9a4595-cc768d49f-fb6l8" podUID="fd3cc6d4-bb20-4c1d-a27f-b623408bafa4" containerName="payload-pre-processing" containerID="cri-o://d57cc647a5acf0d4dc1fdbfaf39d66e0cfc2d82b3a7982df3296a3cec576e614" gracePeriod=25 Nov 02 01:44:03.547915 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:03.547766 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-9a4595-5d57f7c66f-5vgsh" podUID="53ae6505-044c-4e0d-8962-7aa6eb262ac1" containerName="payload-processing" containerID="cri-o://c6e7df11e1809ac34f6e438c3e7925fa5b8f9b2b74353a2fe2a8ffdf9c80b0b6" gracePeriod=25 Nov 02 01:44:03.576552 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:03.576038 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ed6780b-6074-4ce4-b9c0-44253b32955c" path="/var/lib/kubelet/pods/0ed6780b-6074-4ce4-b9c0-44253b32955c/volumes" Nov 02 01:44:03.578849 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:03.578820 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23707aeb-a006-4049-bca4-074bbd446ca7" path="/var/lib/kubelet/pods/23707aeb-a006-4049-bca4-074bbd446ca7/volumes" Nov 02 01:44:03.580162 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:03.580030 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57445e45-2af1-475b-a32b-e8d88a5cdc54" path="/var/lib/kubelet/pods/57445e45-2af1-475b-a32b-e8d88a5cdc54/volumes" Nov 02 01:44:03.586691 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:03.581937 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1412019-62bb-4fc3-a7ad-8e123fef90eb" path="/var/lib/kubelet/pods/e1412019-62bb-4fc3-a7ad-8e123fef90eb/volumes" Nov 02 01:44:03.672184 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:03.671835 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-ffe5c1-579789c787-hz68t" podUID="b9bb7aa5-03de-42b9-a6d3-b1d8e7806a74" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 01:44:03.831417 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:03.830779 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g" Nov 02 01:44:03.927979 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:03.927953 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9a4595-5d57f7c66f-5vgsh" Nov 02 01:44:03.971391 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:03.971265 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b8221ff-cb53-42c0-a697-4992c40d8e52-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b8221ff-cb53-42c0-a697-4992c40d8e52-workload-certs\") pod \"6b8221ff-cb53-42c0-a697-4992c40d8e52\" (UID: \"6b8221ff-cb53-42c0-a697-4992c40d8e52\") " Nov 02 01:44:03.971868 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:03.971448 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b8221ff-cb53-42c0-a697-4992c40d8e52-istio-token\" (UniqueName: \"kubernetes.io/projected/6b8221ff-cb53-42c0-a697-4992c40d8e52-istio-token\") pod \"6b8221ff-cb53-42c0-a697-4992c40d8e52\" (UID: \"6b8221ff-cb53-42c0-a697-4992c40d8e52\") " Nov 02 01:44:03.971868 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:03.971494 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53ae6505-044c-4e0d-8962-7aa6eb262ac1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53ae6505-044c-4e0d-8962-7aa6eb262ac1-plugins-config-volume\") pod \"53ae6505-044c-4e0d-8962-7aa6eb262ac1\" (UID: \"53ae6505-044c-4e0d-8962-7aa6eb262ac1\") " Nov 02 01:44:03.971868 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:03.971592 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b8221ff-cb53-42c0-a697-4992c40d8e52-workload-certs" pod "6b8221ff-cb53-42c0-a697-4992c40d8e52" (UID: "6b8221ff-cb53-42c0-a697-4992c40d8e52"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:03.972118 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:03.972018 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53ae6505-044c-4e0d-8962-7aa6eb262ac1-plugins-config-volume" pod "53ae6505-044c-4e0d-8962-7aa6eb262ac1" (UID: "53ae6505-044c-4e0d-8962-7aa6eb262ac1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:44:03.972941 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:03.972443 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b8221ff-cb53-42c0-a697-4992c40d8e52-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b8221ff-cb53-42c0-a697-4992c40d8e52-istiod-ca-cert\") pod \"6b8221ff-cb53-42c0-a697-4992c40d8e52\" (UID: \"6b8221ff-cb53-42c0-a697-4992c40d8e52\") " Nov 02 01:44:03.972941 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:03.972581 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b8221ff-cb53-42c0-a697-4992c40d8e52-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b8221ff-cb53-42c0-a697-4992c40d8e52-workload-socket\") pod \"6b8221ff-cb53-42c0-a697-4992c40d8e52\" (UID: \"6b8221ff-cb53-42c0-a697-4992c40d8e52\") " Nov 02 01:44:03.972941 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:03.972620 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b8221ff-cb53-42c0-a697-4992c40d8e52-kube-api-access-jrx4s\" (UniqueName: \"kubernetes.io/projected/6b8221ff-cb53-42c0-a697-4992c40d8e52-kube-api-access-jrx4s\") pod \"6b8221ff-cb53-42c0-a697-4992c40d8e52\" (UID: \"6b8221ff-cb53-42c0-a697-4992c40d8e52\") " Nov 02 01:44:03.972941 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:03.972660 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b8221ff-cb53-42c0-a697-4992c40d8e52-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b8221ff-cb53-42c0-a697-4992c40d8e52-istio-data\") pod \"6b8221ff-cb53-42c0-a697-4992c40d8e52\" (UID: \"6b8221ff-cb53-42c0-a697-4992c40d8e52\") " Nov 02 01:44:03.972941 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:03.972745 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b8221ff-cb53-42c0-a697-4992c40d8e52-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b8221ff-cb53-42c0-a697-4992c40d8e52-istio-envoy\") pod \"6b8221ff-cb53-42c0-a697-4992c40d8e52\" (UID: \"6b8221ff-cb53-42c0-a697-4992c40d8e52\") " Nov 02 01:44:03.972941 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:03.972829 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b8221ff-cb53-42c0-a697-4992c40d8e52-istiod-ca-cert" pod "6b8221ff-cb53-42c0-a697-4992c40d8e52" (UID: "6b8221ff-cb53-42c0-a697-4992c40d8e52"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:44:03.972941 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:03.972849 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b8221ff-cb53-42c0-a697-4992c40d8e52-workload-socket" pod "6b8221ff-cb53-42c0-a697-4992c40d8e52" (UID: "6b8221ff-cb53-42c0-a697-4992c40d8e52"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:03.972941 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:03.972856 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b8221ff-cb53-42c0-a697-4992c40d8e52-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b8221ff-cb53-42c0-a697-4992c40d8e52-istio-podinfo\") pod \"6b8221ff-cb53-42c0-a697-4992c40d8e52\" (UID: \"6b8221ff-cb53-42c0-a697-4992c40d8e52\") " Nov 02 01:44:03.973393 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:03.972991 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53ae6505-044c-4e0d-8962-7aa6eb262ac1-kube-api-access-xs9fz\" (UniqueName: \"kubernetes.io/projected/53ae6505-044c-4e0d-8962-7aa6eb262ac1-kube-api-access-xs9fz\") pod \"53ae6505-044c-4e0d-8962-7aa6eb262ac1\" (UID: \"53ae6505-044c-4e0d-8962-7aa6eb262ac1\") " Nov 02 01:44:03.973393 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:03.973067 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b8221ff-cb53-42c0-a697-4992c40d8e52-istio-data" pod "6b8221ff-cb53-42c0-a697-4992c40d8e52" (UID: "6b8221ff-cb53-42c0-a697-4992c40d8e52"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:03.973524 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:03.973117 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b8221ff-cb53-42c0-a697-4992c40d8e52-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b8221ff-cb53-42c0-a697-4992c40d8e52-credential-socket\") pod \"6b8221ff-cb53-42c0-a697-4992c40d8e52\" (UID: \"6b8221ff-cb53-42c0-a697-4992c40d8e52\") " Nov 02 01:44:03.974733 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:03.974203 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b8221ff-cb53-42c0-a697-4992c40d8e52-credential-socket" pod "6b8221ff-cb53-42c0-a697-4992c40d8e52" (UID: "6b8221ff-cb53-42c0-a697-4992c40d8e52"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:03.974733 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:03.974626 4427 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b8221ff-cb53-42c0-a697-4992c40d8e52-credential-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:03.974733 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:03.974651 4427 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b8221ff-cb53-42c0-a697-4992c40d8e52-workload-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:03.974733 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:03.974666 4427 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53ae6505-044c-4e0d-8962-7aa6eb262ac1-plugins-config-volume\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:03.974733 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:03.974682 4427 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b8221ff-cb53-42c0-a697-4992c40d8e52-istiod-ca-cert\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:03.974733 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:03.974695 4427 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b8221ff-cb53-42c0-a697-4992c40d8e52-workload-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:03.974733 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:03.974709 4427 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b8221ff-cb53-42c0-a697-4992c40d8e52-istio-data\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:03.975637 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:03.975600 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b8221ff-cb53-42c0-a697-4992c40d8e52-istio-token" pod "6b8221ff-cb53-42c0-a697-4992c40d8e52" (UID: "6b8221ff-cb53-42c0-a697-4992c40d8e52"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:44:03.977252 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:03.977212 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53ae6505-044c-4e0d-8962-7aa6eb262ac1-kube-api-access-xs9fz" pod "53ae6505-044c-4e0d-8962-7aa6eb262ac1" (UID: "53ae6505-044c-4e0d-8962-7aa6eb262ac1"). InnerVolumeSpecName "kube-api-access-xs9fz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:44:03.977525 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:03.977499 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b8221ff-cb53-42c0-a697-4992c40d8e52-kube-api-access-jrx4s" pod "6b8221ff-cb53-42c0-a697-4992c40d8e52" (UID: "6b8221ff-cb53-42c0-a697-4992c40d8e52"). InnerVolumeSpecName "kube-api-access-jrx4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:44:03.977778 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:03.977757 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b8221ff-cb53-42c0-a697-4992c40d8e52-istio-podinfo" pod "6b8221ff-cb53-42c0-a697-4992c40d8e52" (UID: "6b8221ff-cb53-42c0-a697-4992c40d8e52"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:44:03.977887 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:03.977842 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b8221ff-cb53-42c0-a697-4992c40d8e52-istio-envoy" pod "6b8221ff-cb53-42c0-a697-4992c40d8e52" (UID: "6b8221ff-cb53-42c0-a697-4992c40d8e52"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:03.981900 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:03.981866 4427 generic.go:356] "Generic (PLEG): container finished" podID="6b8221ff-cb53-42c0-a697-4992c40d8e52" containerID="1758ac1393ccd3d867b999af3e71362b1a13d3afb4e9c7a6a6da7c8c16705754" exitCode=0 Nov 02 01:44:03.982005 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:03.981915 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g" Nov 02 01:44:03.982005 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:03.981940 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g" event={"ID":"6b8221ff-cb53-42c0-a697-4992c40d8e52","Type":"ContainerDied","Data":"1758ac1393ccd3d867b999af3e71362b1a13d3afb4e9c7a6a6da7c8c16705754"} Nov 02 01:44:03.982005 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:03.981996 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g" event={"ID":"6b8221ff-cb53-42c0-a697-4992c40d8e52","Type":"ContainerDied","Data":"7ff643b96a21b245a13c9aeebcc61bfccd50ac0498a83af7281f4725420c482f"} Nov 02 01:44:03.982168 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:03.982026 4427 scope.go:122] "RemoveContainer" containerID="1758ac1393ccd3d867b999af3e71362b1a13d3afb4e9c7a6a6da7c8c16705754" Nov 02 01:44:03.984099 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:03.983832 4427 generic.go:356] "Generic (PLEG): container finished" podID="53ae6505-044c-4e0d-8962-7aa6eb262ac1" containerID="c6e7df11e1809ac34f6e438c3e7925fa5b8f9b2b74353a2fe2a8ffdf9c80b0b6" exitCode=0 Nov 02 01:44:03.984099 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:03.984029 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9a4595-5d57f7c66f-5vgsh" event={"ID":"53ae6505-044c-4e0d-8962-7aa6eb262ac1","Type":"ContainerDied","Data":"c6e7df11e1809ac34f6e438c3e7925fa5b8f9b2b74353a2fe2a8ffdf9c80b0b6"} Nov 02 01:44:03.984099 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:03.984084 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9a4595-5d57f7c66f-5vgsh" event={"ID":"53ae6505-044c-4e0d-8962-7aa6eb262ac1","Type":"ContainerDied","Data":"891cef75aca2ce081b49aa940f042a1783d60a2318576ff41b16e4dbb2175c9b"} Nov 02 01:44:03.984281 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:03.984042 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9a4595-5d57f7c66f-5vgsh" Nov 02 01:44:04.007055 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:04.007024 4427 scope.go:122] "RemoveContainer" containerID="1758ac1393ccd3d867b999af3e71362b1a13d3afb4e9c7a6a6da7c8c16705754" Nov 02 01:44:04.007450 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:44:04.007429 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1758ac1393ccd3d867b999af3e71362b1a13d3afb4e9c7a6a6da7c8c16705754\": container with ID starting with 1758ac1393ccd3d867b999af3e71362b1a13d3afb4e9c7a6a6da7c8c16705754 not found: ID does not exist" containerID="1758ac1393ccd3d867b999af3e71362b1a13d3afb4e9c7a6a6da7c8c16705754" Nov 02 01:44:04.007533 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:04.007461 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1758ac1393ccd3d867b999af3e71362b1a13d3afb4e9c7a6a6da7c8c16705754"} err="failed to get container status \"1758ac1393ccd3d867b999af3e71362b1a13d3afb4e9c7a6a6da7c8c16705754\": rpc error: code = NotFound desc = could not find container \"1758ac1393ccd3d867b999af3e71362b1a13d3afb4e9c7a6a6da7c8c16705754\": container with ID starting with 1758ac1393ccd3d867b999af3e71362b1a13d3afb4e9c7a6a6da7c8c16705754 not found: ID does not exist" Nov 02 01:44:04.007533 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:04.007494 4427 scope.go:122] "RemoveContainer" containerID="c6e7df11e1809ac34f6e438c3e7925fa5b8f9b2b74353a2fe2a8ffdf9c80b0b6" Nov 02 01:44:04.018213 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:04.018191 4427 scope.go:122] "RemoveContainer" containerID="c6e7df11e1809ac34f6e438c3e7925fa5b8f9b2b74353a2fe2a8ffdf9c80b0b6" Nov 02 01:44:04.018607 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:44:04.018582 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6e7df11e1809ac34f6e438c3e7925fa5b8f9b2b74353a2fe2a8ffdf9c80b0b6\": container with ID starting with c6e7df11e1809ac34f6e438c3e7925fa5b8f9b2b74353a2fe2a8ffdf9c80b0b6 not found: ID does not exist" containerID="c6e7df11e1809ac34f6e438c3e7925fa5b8f9b2b74353a2fe2a8ffdf9c80b0b6" Nov 02 01:44:04.018708 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:04.018616 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6e7df11e1809ac34f6e438c3e7925fa5b8f9b2b74353a2fe2a8ffdf9c80b0b6"} err="failed to get container status \"c6e7df11e1809ac34f6e438c3e7925fa5b8f9b2b74353a2fe2a8ffdf9c80b0b6\": rpc error: code = NotFound desc = could not find container \"c6e7df11e1809ac34f6e438c3e7925fa5b8f9b2b74353a2fe2a8ffdf9c80b0b6\": container with ID starting with c6e7df11e1809ac34f6e438c3e7925fa5b8f9b2b74353a2fe2a8ffdf9c80b0b6 not found: ID does not exist" Nov 02 01:44:04.022646 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:04.022619 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g"] Nov 02 01:44:04.028624 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:04.028602 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-2589b2-openshift-default-5776fd4969-jms9g"] Nov 02 01:44:04.040581 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:04.040547 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9a4595-5d57f7c66f-5vgsh"] Nov 02 01:44:04.044694 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:04.044669 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9a4595-5d57f7c66f-5vgsh"] Nov 02 01:44:04.076940 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:04.076912 4427 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b8221ff-cb53-42c0-a697-4992c40d8e52-istio-token\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:04.076940 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:04.076939 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrx4s\" (UniqueName: \"kubernetes.io/projected/6b8221ff-cb53-42c0-a697-4992c40d8e52-kube-api-access-jrx4s\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:04.077129 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:04.076953 4427 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b8221ff-cb53-42c0-a697-4992c40d8e52-istio-envoy\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:04.077129 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:04.076963 4427 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b8221ff-cb53-42c0-a697-4992c40d8e52-istio-podinfo\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:04.077129 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:04.076972 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs9fz\" (UniqueName: \"kubernetes.io/projected/53ae6505-044c-4e0d-8962-7aa6eb262ac1-kube-api-access-xs9fz\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:05.622279 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:05.622162 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53ae6505-044c-4e0d-8962-7aa6eb262ac1" path="/var/lib/kubelet/pods/53ae6505-044c-4e0d-8962-7aa6eb262ac1/volumes" Nov 02 01:44:05.623868 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:05.623843 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b8221ff-cb53-42c0-a697-4992c40d8e52" path="/var/lib/kubelet/pods/6b8221ff-cb53-42c0-a697-4992c40d8e52/volumes" Nov 02 01:44:05.686428 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:05.682737 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r"] Nov 02 01:44:05.686428 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:05.682947 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r" podUID="c227e450-ed88-4cb4-a5fd-e88f150d14ab" containerName="istio-proxy" containerID="cri-o://ec2824c66fba034da4af5829a533e93ac6210f64fcc303b7204058fb3889ef4e" gracePeriod=30 Nov 02 01:44:05.718355 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:05.718312 4427 patch_prober.go:29] interesting pod/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 01:44:05.718601 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:05.718405 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r" podUID="c227e450-ed88-4cb4-a5fd-e88f150d14ab" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 01:44:06.827421 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:06.827388 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2589b2-5b4dc6cc9f-nmtkc" Nov 02 01:44:06.864886 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:06.864799 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a097a1f1-7f03-479d-a65a-90f8e542d06b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a097a1f1-7f03-479d-a65a-90f8e542d06b-plugins-config-volume\") pod \"a097a1f1-7f03-479d-a65a-90f8e542d06b\" (UID: \"a097a1f1-7f03-479d-a65a-90f8e542d06b\") " Nov 02 01:44:06.865084 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:06.864991 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a097a1f1-7f03-479d-a65a-90f8e542d06b-kube-api-access-669b8\" (UniqueName: \"kubernetes.io/projected/a097a1f1-7f03-479d-a65a-90f8e542d06b-kube-api-access-669b8\") pod \"a097a1f1-7f03-479d-a65a-90f8e542d06b\" (UID: \"a097a1f1-7f03-479d-a65a-90f8e542d06b\") " Nov 02 01:44:06.865608 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:06.865584 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a097a1f1-7f03-479d-a65a-90f8e542d06b-plugins-config-volume" pod "a097a1f1-7f03-479d-a65a-90f8e542d06b" (UID: "a097a1f1-7f03-479d-a65a-90f8e542d06b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:44:06.867900 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:06.867870 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a097a1f1-7f03-479d-a65a-90f8e542d06b-kube-api-access-669b8" pod "a097a1f1-7f03-479d-a65a-90f8e542d06b" (UID: "a097a1f1-7f03-479d-a65a-90f8e542d06b"). InnerVolumeSpecName "kube-api-access-669b8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:44:06.966970 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:06.966931 4427 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a097a1f1-7f03-479d-a65a-90f8e542d06b-plugins-config-volume\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:06.966970 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:06.966966 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-669b8\" (UniqueName: \"kubernetes.io/projected/a097a1f1-7f03-479d-a65a-90f8e542d06b-kube-api-access-669b8\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:07.041314 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:07.041275 4427 generic.go:356] "Generic (PLEG): container finished" podID="a097a1f1-7f03-479d-a65a-90f8e542d06b" containerID="b99b75d2d5d0e26802f6fa73863ea110cfe140e58ceca80294d507733ca008e8" exitCode=0 Nov 02 01:44:07.041583 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:07.041324 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2589b2-5b4dc6cc9f-nmtkc" Nov 02 01:44:07.041583 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:07.041350 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2589b2-5b4dc6cc9f-nmtkc" event={"ID":"a097a1f1-7f03-479d-a65a-90f8e542d06b","Type":"ContainerDied","Data":"b99b75d2d5d0e26802f6fa73863ea110cfe140e58ceca80294d507733ca008e8"} Nov 02 01:44:07.041583 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:07.041457 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2589b2-5b4dc6cc9f-nmtkc" event={"ID":"a097a1f1-7f03-479d-a65a-90f8e542d06b","Type":"ContainerDied","Data":"b13291f9539f24f740c82745c97e264a7139399ba3aa1eeac3bc8222cd545d07"} Nov 02 01:44:07.041583 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:07.041492 4427 scope.go:122] "RemoveContainer" containerID="b99b75d2d5d0e26802f6fa73863ea110cfe140e58ceca80294d507733ca008e8" Nov 02 01:44:07.054564 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:07.054537 4427 scope.go:122] "RemoveContainer" containerID="b99b75d2d5d0e26802f6fa73863ea110cfe140e58ceca80294d507733ca008e8" Nov 02 01:44:07.054863 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:44:07.054841 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b99b75d2d5d0e26802f6fa73863ea110cfe140e58ceca80294d507733ca008e8\": container with ID starting with b99b75d2d5d0e26802f6fa73863ea110cfe140e58ceca80294d507733ca008e8 not found: ID does not exist" containerID="b99b75d2d5d0e26802f6fa73863ea110cfe140e58ceca80294d507733ca008e8" Nov 02 01:44:07.054944 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:07.054870 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b99b75d2d5d0e26802f6fa73863ea110cfe140e58ceca80294d507733ca008e8"} err="failed to get container status \"b99b75d2d5d0e26802f6fa73863ea110cfe140e58ceca80294d507733ca008e8\": rpc error: code = NotFound desc = could not find container \"b99b75d2d5d0e26802f6fa73863ea110cfe140e58ceca80294d507733ca008e8\": container with ID starting with b99b75d2d5d0e26802f6fa73863ea110cfe140e58ceca80294d507733ca008e8 not found: ID does not exist" Nov 02 01:44:07.146919 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:07.146822 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2589b2-5b4dc6cc9f-nmtkc"] Nov 02 01:44:07.151087 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:07.151059 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2589b2-5b4dc6cc9f-nmtkc"] Nov 02 01:44:07.586450 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:07.586033 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a097a1f1-7f03-479d-a65a-90f8e542d06b" path="/var/lib/kubelet/pods/a097a1f1-7f03-479d-a65a-90f8e542d06b/volumes" Nov 02 01:44:10.727442 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:10.726598 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d"] Nov 02 01:44:10.727442 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:10.726888 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d" podUID="f6da871e-dedf-412b-bbad-4257087b71a7" containerName="istio-proxy" containerID="cri-o://0c04a2ada11dc080b9a1951ef18087dfc06e5e0b3e3b749d6fcac3e47c183669" gracePeriod=30 Nov 02 01:44:11.036461 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:10.998982 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7"] Nov 02 01:44:11.036461 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:10.999275 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7" podUID="273e0a10-0ba6-41bb-b8db-1ff672905957" containerName="istio-proxy" containerID="cri-o://77854ebe0744657b26b6612e13f07e1bb749c776817ca86199013bbf26894742" gracePeriod=30 Nov 02 01:44:11.354451 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:11.353816 4427 generic.go:356] "Generic (PLEG): container finished" podID="c227e450-ed88-4cb4-a5fd-e88f150d14ab" containerID="ec2824c66fba034da4af5829a533e93ac6210f64fcc303b7204058fb3889ef4e" exitCode=0 Nov 02 01:44:11.354451 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:11.353960 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r" event={"ID":"c227e450-ed88-4cb4-a5fd-e88f150d14ab","Type":"ContainerDied","Data":"ec2824c66fba034da4af5829a533e93ac6210f64fcc303b7204058fb3889ef4e"} Nov 02 01:44:11.354451 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:11.354027 4427 scope.go:122] "RemoveContainer" containerID="b572c45e63c2baf75b3cbe98f904258edf95b1dde3445abb8a01ea161ae61022" Nov 02 01:44:11.585474 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:11.578412 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9a4595-cc768d49f-fb6l8" podUID="fd3cc6d4-bb20-4c1d-a27f-b623408bafa4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 01:44:11.852981 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:11.852947 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r" Nov 02 01:44:11.954303 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:11.954204 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c227e450-ed88-4cb4-a5fd-e88f150d14ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c227e450-ed88-4cb4-a5fd-e88f150d14ab-workload-certs\") pod \"c227e450-ed88-4cb4-a5fd-e88f150d14ab\" (UID: \"c227e450-ed88-4cb4-a5fd-e88f150d14ab\") " Nov 02 01:44:11.954497 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:11.954351 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c227e450-ed88-4cb4-a5fd-e88f150d14ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c227e450-ed88-4cb4-a5fd-e88f150d14ab-istiod-ca-cert\") pod \"c227e450-ed88-4cb4-a5fd-e88f150d14ab\" (UID: \"c227e450-ed88-4cb4-a5fd-e88f150d14ab\") " Nov 02 01:44:11.954497 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:11.954475 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c227e450-ed88-4cb4-a5fd-e88f150d14ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c227e450-ed88-4cb4-a5fd-e88f150d14ab-istio-data\") pod \"c227e450-ed88-4cb4-a5fd-e88f150d14ab\" (UID: \"c227e450-ed88-4cb4-a5fd-e88f150d14ab\") " Nov 02 01:44:11.954668 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:11.954536 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c227e450-ed88-4cb4-a5fd-e88f150d14ab-kube-api-access-657hs\" (UniqueName: \"kubernetes.io/projected/c227e450-ed88-4cb4-a5fd-e88f150d14ab-kube-api-access-657hs\") pod \"c227e450-ed88-4cb4-a5fd-e88f150d14ab\" (UID: \"c227e450-ed88-4cb4-a5fd-e88f150d14ab\") " Nov 02 01:44:11.954668 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:11.954606 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c227e450-ed88-4cb4-a5fd-e88f150d14ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c227e450-ed88-4cb4-a5fd-e88f150d14ab-workload-socket\") pod \"c227e450-ed88-4cb4-a5fd-e88f150d14ab\" (UID: \"c227e450-ed88-4cb4-a5fd-e88f150d14ab\") " Nov 02 01:44:11.954777 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:11.954533 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c227e450-ed88-4cb4-a5fd-e88f150d14ab-workload-certs" pod "c227e450-ed88-4cb4-a5fd-e88f150d14ab" (UID: "c227e450-ed88-4cb4-a5fd-e88f150d14ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:11.954777 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:11.954710 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c227e450-ed88-4cb4-a5fd-e88f150d14ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c227e450-ed88-4cb4-a5fd-e88f150d14ab-istio-envoy\") pod \"c227e450-ed88-4cb4-a5fd-e88f150d14ab\" (UID: \"c227e450-ed88-4cb4-a5fd-e88f150d14ab\") " Nov 02 01:44:11.954777 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:11.954766 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c227e450-ed88-4cb4-a5fd-e88f150d14ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c227e450-ed88-4cb4-a5fd-e88f150d14ab-credential-socket\") pod \"c227e450-ed88-4cb4-a5fd-e88f150d14ab\" (UID: \"c227e450-ed88-4cb4-a5fd-e88f150d14ab\") " Nov 02 01:44:11.954932 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:11.954772 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c227e450-ed88-4cb4-a5fd-e88f150d14ab-istio-data" pod "c227e450-ed88-4cb4-a5fd-e88f150d14ab" (UID: "c227e450-ed88-4cb4-a5fd-e88f150d14ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:11.954932 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:11.954818 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c227e450-ed88-4cb4-a5fd-e88f150d14ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c227e450-ed88-4cb4-a5fd-e88f150d14ab-istio-podinfo\") pod \"c227e450-ed88-4cb4-a5fd-e88f150d14ab\" (UID: \"c227e450-ed88-4cb4-a5fd-e88f150d14ab\") " Nov 02 01:44:11.954932 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:11.954832 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c227e450-ed88-4cb4-a5fd-e88f150d14ab-istiod-ca-cert" pod "c227e450-ed88-4cb4-a5fd-e88f150d14ab" (UID: "c227e450-ed88-4cb4-a5fd-e88f150d14ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:44:11.954932 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:11.954864 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c227e450-ed88-4cb4-a5fd-e88f150d14ab-istio-token\" (UniqueName: \"kubernetes.io/projected/c227e450-ed88-4cb4-a5fd-e88f150d14ab-istio-token\") pod \"c227e450-ed88-4cb4-a5fd-e88f150d14ab\" (UID: \"c227e450-ed88-4cb4-a5fd-e88f150d14ab\") " Nov 02 01:44:11.955143 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:11.955008 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c227e450-ed88-4cb4-a5fd-e88f150d14ab-workload-socket" pod "c227e450-ed88-4cb4-a5fd-e88f150d14ab" (UID: "c227e450-ed88-4cb4-a5fd-e88f150d14ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:11.955203 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:11.955188 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c227e450-ed88-4cb4-a5fd-e88f150d14ab-credential-socket" pod "c227e450-ed88-4cb4-a5fd-e88f150d14ab" (UID: "c227e450-ed88-4cb4-a5fd-e88f150d14ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:11.955657 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:11.955631 4427 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c227e450-ed88-4cb4-a5fd-e88f150d14ab-credential-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:11.955848 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:11.955825 4427 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c227e450-ed88-4cb4-a5fd-e88f150d14ab-workload-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:11.955973 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:11.955851 4427 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c227e450-ed88-4cb4-a5fd-e88f150d14ab-istiod-ca-cert\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:11.955973 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:11.955867 4427 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c227e450-ed88-4cb4-a5fd-e88f150d14ab-istio-data\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:11.955973 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:11.955880 4427 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c227e450-ed88-4cb4-a5fd-e88f150d14ab-workload-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:11.963429 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:11.958591 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c227e450-ed88-4cb4-a5fd-e88f150d14ab-istio-podinfo" pod "c227e450-ed88-4cb4-a5fd-e88f150d14ab" (UID: "c227e450-ed88-4cb4-a5fd-e88f150d14ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:44:11.972913 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:11.972875 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c227e450-ed88-4cb4-a5fd-e88f150d14ab-istio-envoy" pod "c227e450-ed88-4cb4-a5fd-e88f150d14ab" (UID: "c227e450-ed88-4cb4-a5fd-e88f150d14ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:11.973120 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:11.972930 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c227e450-ed88-4cb4-a5fd-e88f150d14ab-kube-api-access-657hs" pod "c227e450-ed88-4cb4-a5fd-e88f150d14ab" (UID: "c227e450-ed88-4cb4-a5fd-e88f150d14ab"). InnerVolumeSpecName "kube-api-access-657hs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:44:11.982350 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:11.982308 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c227e450-ed88-4cb4-a5fd-e88f150d14ab-istio-token" pod "c227e450-ed88-4cb4-a5fd-e88f150d14ab" (UID: "c227e450-ed88-4cb4-a5fd-e88f150d14ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:44:12.057833 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:12.057795 4427 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c227e450-ed88-4cb4-a5fd-e88f150d14ab-istio-podinfo\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:12.057833 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:12.057830 4427 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c227e450-ed88-4cb4-a5fd-e88f150d14ab-istio-token\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:12.058048 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:12.057843 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-657hs\" (UniqueName: \"kubernetes.io/projected/c227e450-ed88-4cb4-a5fd-e88f150d14ab-kube-api-access-657hs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:12.058048 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:12.057857 4427 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c227e450-ed88-4cb4-a5fd-e88f150d14ab-istio-envoy\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:12.366391 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:12.366336 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r" event={"ID":"c227e450-ed88-4cb4-a5fd-e88f150d14ab","Type":"ContainerDied","Data":"1ae611910ee496162f4a5c4e6c381fd4fb6c926fa36fc442de326517682b31e4"} Nov 02 01:44:12.366577 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:12.366431 4427 scope.go:122] "RemoveContainer" containerID="ec2824c66fba034da4af5829a533e93ac6210f64fcc303b7204058fb3889ef4e" Nov 02 01:44:12.366577 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:12.366548 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r" Nov 02 01:44:12.403811 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:12.403777 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r"] Nov 02 01:44:12.409757 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:12.409722 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-ffe5c1-openshift-default-5b9f64955c-qdp4r"] Nov 02 01:44:13.539508 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:13.539468 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c227e450-ed88-4cb4-a5fd-e88f150d14ab" path="/var/lib/kubelet/pods/c227e450-ed88-4cb4-a5fd-e88f150d14ab/volumes" Nov 02 01:44:13.822468 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:13.822432 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ffe5c1-579789c787-hz68t" Nov 02 01:44:13.950689 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:13.950653 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9bb7aa5-03de-42b9-a6d3-b1d8e7806a74-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9bb7aa5-03de-42b9-a6d3-b1d8e7806a74-plugins-config-volume\") pod \"b9bb7aa5-03de-42b9-a6d3-b1d8e7806a74\" (UID: \"b9bb7aa5-03de-42b9-a6d3-b1d8e7806a74\") " Nov 02 01:44:13.950887 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:13.950719 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9bb7aa5-03de-42b9-a6d3-b1d8e7806a74-kube-api-access-572r8\" (UniqueName: \"kubernetes.io/projected/b9bb7aa5-03de-42b9-a6d3-b1d8e7806a74-kube-api-access-572r8\") pod \"b9bb7aa5-03de-42b9-a6d3-b1d8e7806a74\" (UID: \"b9bb7aa5-03de-42b9-a6d3-b1d8e7806a74\") " Nov 02 01:44:13.951076 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:13.951052 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9bb7aa5-03de-42b9-a6d3-b1d8e7806a74-plugins-config-volume" pod "b9bb7aa5-03de-42b9-a6d3-b1d8e7806a74" (UID: "b9bb7aa5-03de-42b9-a6d3-b1d8e7806a74"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:44:13.951680 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:13.951652 4427 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9bb7aa5-03de-42b9-a6d3-b1d8e7806a74-plugins-config-volume\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:13.953212 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:13.953189 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9bb7aa5-03de-42b9-a6d3-b1d8e7806a74-kube-api-access-572r8" pod "b9bb7aa5-03de-42b9-a6d3-b1d8e7806a74" (UID: "b9bb7aa5-03de-42b9-a6d3-b1d8e7806a74"). InnerVolumeSpecName "kube-api-access-572r8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:44:14.054031 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:14.053903 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-572r8\" (UniqueName: \"kubernetes.io/projected/b9bb7aa5-03de-42b9-a6d3-b1d8e7806a74-kube-api-access-572r8\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:14.249916 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:14.249880 4427 patch_prober.go:29] interesting pod/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 01:44:14.250072 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:14.249939 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d" podUID="f6da871e-dedf-412b-bbad-4257087b71a7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 01:44:14.389244 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:14.389204 4427 generic.go:356] "Generic (PLEG): container finished" podID="b9bb7aa5-03de-42b9-a6d3-b1d8e7806a74" containerID="de6f1c505982666c93a25403b9eea087c169cd8dac0531c888ef4a3351663a60" exitCode=0 Nov 02 01:44:14.389464 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:14.389305 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ffe5c1-579789c787-hz68t" Nov 02 01:44:14.389464 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:14.389329 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ffe5c1-579789c787-hz68t" event={"ID":"b9bb7aa5-03de-42b9-a6d3-b1d8e7806a74","Type":"ContainerDied","Data":"de6f1c505982666c93a25403b9eea087c169cd8dac0531c888ef4a3351663a60"} Nov 02 01:44:14.389464 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:14.389440 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ffe5c1-579789c787-hz68t" event={"ID":"b9bb7aa5-03de-42b9-a6d3-b1d8e7806a74","Type":"ContainerDied","Data":"339373e483955ef88e515446956fe0ba62bb5cdf18349e2a6f98b7da721c0d47"} Nov 02 01:44:14.389712 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:14.389473 4427 scope.go:122] "RemoveContainer" containerID="de6f1c505982666c93a25403b9eea087c169cd8dac0531c888ef4a3351663a60" Nov 02 01:44:14.404597 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:14.404566 4427 scope.go:122] "RemoveContainer" containerID="de6f1c505982666c93a25403b9eea087c169cd8dac0531c888ef4a3351663a60" Nov 02 01:44:14.405083 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:44:14.405048 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de6f1c505982666c93a25403b9eea087c169cd8dac0531c888ef4a3351663a60\": container with ID starting with de6f1c505982666c93a25403b9eea087c169cd8dac0531c888ef4a3351663a60 not found: ID does not exist" containerID="de6f1c505982666c93a25403b9eea087c169cd8dac0531c888ef4a3351663a60" Nov 02 01:44:14.405189 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:14.405094 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de6f1c505982666c93a25403b9eea087c169cd8dac0531c888ef4a3351663a60"} err="failed to get container status \"de6f1c505982666c93a25403b9eea087c169cd8dac0531c888ef4a3351663a60\": rpc error: code = NotFound desc = could not find container \"de6f1c505982666c93a25403b9eea087c169cd8dac0531c888ef4a3351663a60\": container with ID starting with de6f1c505982666c93a25403b9eea087c169cd8dac0531c888ef4a3351663a60 not found: ID does not exist" Nov 02 01:44:14.421680 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:14.421655 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ffe5c1-579789c787-hz68t"] Nov 02 01:44:14.425610 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:14.425589 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ffe5c1-579789c787-hz68t"] Nov 02 01:44:14.670115 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:14.670006 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-ffe5c1-579789c787-hz68t" podUID="b9bb7aa5-03de-42b9-a6d3-b1d8e7806a74" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.111:50052\" within 1s: context deadline exceeded" Nov 02 01:44:14.670115 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:44:14.670104 4427 logging.go:55] [core] [Channel #2996 SubChannel #2997]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.111:50052", ServerName: "10.133.0.111:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.111:50052: operation was canceled" Nov 02 01:44:15.535702 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:15.535664 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9bb7aa5-03de-42b9-a6d3-b1d8e7806a74" path="/var/lib/kubelet/pods/b9bb7aa5-03de-42b9-a6d3-b1d8e7806a74/volumes" Nov 02 01:44:16.031598 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:16.031570 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d" Nov 02 01:44:16.207358 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:16.199749 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6da871e-dedf-412b-bbad-4257087b71a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6da871e-dedf-412b-bbad-4257087b71a7-workload-certs\") pod \"f6da871e-dedf-412b-bbad-4257087b71a7\" (UID: \"f6da871e-dedf-412b-bbad-4257087b71a7\") " Nov 02 01:44:16.207358 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:16.199840 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6da871e-dedf-412b-bbad-4257087b71a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6da871e-dedf-412b-bbad-4257087b71a7-istio-data\") pod \"f6da871e-dedf-412b-bbad-4257087b71a7\" (UID: \"f6da871e-dedf-412b-bbad-4257087b71a7\") " Nov 02 01:44:16.207358 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:16.199875 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6da871e-dedf-412b-bbad-4257087b71a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6da871e-dedf-412b-bbad-4257087b71a7-workload-socket\") pod \"f6da871e-dedf-412b-bbad-4257087b71a7\" (UID: \"f6da871e-dedf-412b-bbad-4257087b71a7\") " Nov 02 01:44:16.207358 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:16.199993 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6da871e-dedf-412b-bbad-4257087b71a7-kube-api-access-dvf22\" (UniqueName: \"kubernetes.io/projected/f6da871e-dedf-412b-bbad-4257087b71a7-kube-api-access-dvf22\") pod \"f6da871e-dedf-412b-bbad-4257087b71a7\" (UID: \"f6da871e-dedf-412b-bbad-4257087b71a7\") " Nov 02 01:44:16.207358 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:16.200033 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6da871e-dedf-412b-bbad-4257087b71a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6da871e-dedf-412b-bbad-4257087b71a7-istio-envoy\") pod \"f6da871e-dedf-412b-bbad-4257087b71a7\" (UID: \"f6da871e-dedf-412b-bbad-4257087b71a7\") " Nov 02 01:44:16.207358 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:16.200135 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f6da871e-dedf-412b-bbad-4257087b71a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6da871e-dedf-412b-bbad-4257087b71a7-istio-podinfo\") pod \"f6da871e-dedf-412b-bbad-4257087b71a7\" (UID: \"f6da871e-dedf-412b-bbad-4257087b71a7\") " Nov 02 01:44:16.207358 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:16.200163 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6da871e-dedf-412b-bbad-4257087b71a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6da871e-dedf-412b-bbad-4257087b71a7-credential-socket\") pod \"f6da871e-dedf-412b-bbad-4257087b71a7\" (UID: \"f6da871e-dedf-412b-bbad-4257087b71a7\") " Nov 02 01:44:16.207358 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:16.200195 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6da871e-dedf-412b-bbad-4257087b71a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6da871e-dedf-412b-bbad-4257087b71a7-istiod-ca-cert\") pod \"f6da871e-dedf-412b-bbad-4257087b71a7\" (UID: \"f6da871e-dedf-412b-bbad-4257087b71a7\") " Nov 02 01:44:16.207358 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:16.200431 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6da871e-dedf-412b-bbad-4257087b71a7-istio-token\" (UniqueName: \"kubernetes.io/projected/f6da871e-dedf-412b-bbad-4257087b71a7-istio-token\") pod \"f6da871e-dedf-412b-bbad-4257087b71a7\" (UID: \"f6da871e-dedf-412b-bbad-4257087b71a7\") " Nov 02 01:44:16.207358 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:16.203289 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6da871e-dedf-412b-bbad-4257087b71a7-istiod-ca-cert" pod "f6da871e-dedf-412b-bbad-4257087b71a7" (UID: "f6da871e-dedf-412b-bbad-4257087b71a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:44:16.207358 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:16.203624 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6da871e-dedf-412b-bbad-4257087b71a7-credential-socket" pod "f6da871e-dedf-412b-bbad-4257087b71a7" (UID: "f6da871e-dedf-412b-bbad-4257087b71a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:16.207358 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:16.203979 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6da871e-dedf-412b-bbad-4257087b71a7-workload-socket" pod "f6da871e-dedf-412b-bbad-4257087b71a7" (UID: "f6da871e-dedf-412b-bbad-4257087b71a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:16.207358 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:16.204001 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6da871e-dedf-412b-bbad-4257087b71a7-istio-data" pod "f6da871e-dedf-412b-bbad-4257087b71a7" (UID: "f6da871e-dedf-412b-bbad-4257087b71a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:16.207358 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:16.204120 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6da871e-dedf-412b-bbad-4257087b71a7-workload-certs" pod "f6da871e-dedf-412b-bbad-4257087b71a7" (UID: "f6da871e-dedf-412b-bbad-4257087b71a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:16.210681 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:16.210607 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6da871e-dedf-412b-bbad-4257087b71a7-istio-token" pod "f6da871e-dedf-412b-bbad-4257087b71a7" (UID: "f6da871e-dedf-412b-bbad-4257087b71a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:44:16.240166 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:16.220798 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6da871e-dedf-412b-bbad-4257087b71a7-istio-envoy" pod "f6da871e-dedf-412b-bbad-4257087b71a7" (UID: "f6da871e-dedf-412b-bbad-4257087b71a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:16.240166 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:16.232689 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f6da871e-dedf-412b-bbad-4257087b71a7-istio-podinfo" pod "f6da871e-dedf-412b-bbad-4257087b71a7" (UID: "f6da871e-dedf-412b-bbad-4257087b71a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:44:16.240166 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:16.234827 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6da871e-dedf-412b-bbad-4257087b71a7-kube-api-access-dvf22" pod "f6da871e-dedf-412b-bbad-4257087b71a7" (UID: "f6da871e-dedf-412b-bbad-4257087b71a7"). InnerVolumeSpecName "kube-api-access-dvf22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:44:16.296834 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:16.295415 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv"] Nov 02 01:44:16.306873 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:16.306834 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7"] Nov 02 01:44:16.309580 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:16.309550 4427 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6da871e-dedf-412b-bbad-4257087b71a7-istio-data\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:16.309580 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:16.309583 4427 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6da871e-dedf-412b-bbad-4257087b71a7-workload-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:16.309788 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:16.309599 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvf22\" (UniqueName: \"kubernetes.io/projected/f6da871e-dedf-412b-bbad-4257087b71a7-kube-api-access-dvf22\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:16.309788 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:16.309613 4427 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6da871e-dedf-412b-bbad-4257087b71a7-istio-envoy\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:16.309788 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:16.309626 4427 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6da871e-dedf-412b-bbad-4257087b71a7-istio-podinfo\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:16.309788 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:16.309640 4427 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6da871e-dedf-412b-bbad-4257087b71a7-credential-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:16.309788 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:16.309654 4427 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6da871e-dedf-412b-bbad-4257087b71a7-istiod-ca-cert\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:16.309788 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:16.309667 4427 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6da871e-dedf-412b-bbad-4257087b71a7-istio-token\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:16.309788 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:16.309680 4427 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6da871e-dedf-412b-bbad-4257087b71a7-workload-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:16.719397 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:16.719089 4427 generic.go:356] "Generic (PLEG): container finished" podID="f6da871e-dedf-412b-bbad-4257087b71a7" containerID="0c04a2ada11dc080b9a1951ef18087dfc06e5e0b3e3b749d6fcac3e47c183669" exitCode=0 Nov 02 01:44:16.719397 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:16.719303 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d" event={"ID":"f6da871e-dedf-412b-bbad-4257087b71a7","Type":"ContainerDied","Data":"0c04a2ada11dc080b9a1951ef18087dfc06e5e0b3e3b749d6fcac3e47c183669"} Nov 02 01:44:16.719397 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:16.719355 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d" event={"ID":"f6da871e-dedf-412b-bbad-4257087b71a7","Type":"ContainerDied","Data":"e1c88751a02899b61415f5f398386bc27b82d486b4cce4e3902af6e34ef86320"} Nov 02 01:44:16.723394 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:16.720191 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d" Nov 02 01:44:16.737280 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:16.733765 4427 scope.go:122] "RemoveContainer" containerID="0c04a2ada11dc080b9a1951ef18087dfc06e5e0b3e3b749d6fcac3e47c183669" Nov 02 01:44:16.783120 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:16.783084 4427 generic.go:356] "Generic (PLEG): container finished" podID="273e0a10-0ba6-41bb-b8db-1ff672905957" containerID="77854ebe0744657b26b6612e13f07e1bb749c776817ca86199013bbf26894742" exitCode=0 Nov 02 01:44:16.783578 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:16.783175 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7" event={"ID":"273e0a10-0ba6-41bb-b8db-1ff672905957","Type":"ContainerDied","Data":"77854ebe0744657b26b6612e13f07e1bb749c776817ca86199013bbf26894742"} Nov 02 01:44:16.796395 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:16.793455 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d"] Nov 02 01:44:16.833689 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:16.832888 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-c22a0d-openshift-default-77fbfb7dc7-2pv6d"] Nov 02 01:44:16.835323 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:16.834871 4427 scope.go:122] "RemoveContainer" containerID="c6f8e2e24dbd17f2d6bae654cf655359df1ac226eef7c6366564fb8f6182f828" Nov 02 01:44:16.876964 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:16.876736 4427 scope.go:122] "RemoveContainer" containerID="0c04a2ada11dc080b9a1951ef18087dfc06e5e0b3e3b749d6fcac3e47c183669" Nov 02 01:44:16.877094 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:44:16.877064 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c04a2ada11dc080b9a1951ef18087dfc06e5e0b3e3b749d6fcac3e47c183669\": container with ID starting with 0c04a2ada11dc080b9a1951ef18087dfc06e5e0b3e3b749d6fcac3e47c183669 not found: ID does not exist" containerID="0c04a2ada11dc080b9a1951ef18087dfc06e5e0b3e3b749d6fcac3e47c183669" Nov 02 01:44:16.877141 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:16.877107 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c04a2ada11dc080b9a1951ef18087dfc06e5e0b3e3b749d6fcac3e47c183669"} err="failed to get container status \"0c04a2ada11dc080b9a1951ef18087dfc06e5e0b3e3b749d6fcac3e47c183669\": rpc error: code = NotFound desc = could not find container \"0c04a2ada11dc080b9a1951ef18087dfc06e5e0b3e3b749d6fcac3e47c183669\": container with ID starting with 0c04a2ada11dc080b9a1951ef18087dfc06e5e0b3e3b749d6fcac3e47c183669 not found: ID does not exist" Nov 02 01:44:16.877141 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:16.877133 4427 scope.go:122] "RemoveContainer" containerID="c6f8e2e24dbd17f2d6bae654cf655359df1ac226eef7c6366564fb8f6182f828" Nov 02 01:44:16.877952 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:44:16.877857 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6f8e2e24dbd17f2d6bae654cf655359df1ac226eef7c6366564fb8f6182f828\": container with ID starting with c6f8e2e24dbd17f2d6bae654cf655359df1ac226eef7c6366564fb8f6182f828 not found: ID does not exist" containerID="c6f8e2e24dbd17f2d6bae654cf655359df1ac226eef7c6366564fb8f6182f828" Nov 02 01:44:16.877952 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:16.877902 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6f8e2e24dbd17f2d6bae654cf655359df1ac226eef7c6366564fb8f6182f828"} err="failed to get container status \"c6f8e2e24dbd17f2d6bae654cf655359df1ac226eef7c6366564fb8f6182f828\": rpc error: code = NotFound desc = could not find container \"c6f8e2e24dbd17f2d6bae654cf655359df1ac226eef7c6366564fb8f6182f828\": container with ID starting with c6f8e2e24dbd17f2d6bae654cf655359df1ac226eef7c6366564fb8f6182f828 not found: ID does not exist" Nov 02 01:44:16.877952 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:16.877924 4427 scope.go:122] "RemoveContainer" containerID="31481e93dfdddb0e8abe4b5f38d64987489513b46bdc3259dcacd8fa44b152ba" Nov 02 01:44:16.878463 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:16.878442 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7" Nov 02 01:44:17.024908 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:17.024116 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/273e0a10-0ba6-41bb-b8db-1ff672905957-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/273e0a10-0ba6-41bb-b8db-1ff672905957-workload-certs\") pod \"273e0a10-0ba6-41bb-b8db-1ff672905957\" (UID: \"273e0a10-0ba6-41bb-b8db-1ff672905957\") " Nov 02 01:44:17.024908 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:17.024197 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/273e0a10-0ba6-41bb-b8db-1ff672905957-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/273e0a10-0ba6-41bb-b8db-1ff672905957-istiod-ca-cert\") pod \"273e0a10-0ba6-41bb-b8db-1ff672905957\" (UID: \"273e0a10-0ba6-41bb-b8db-1ff672905957\") " Nov 02 01:44:17.024908 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:17.024271 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/273e0a10-0ba6-41bb-b8db-1ff672905957-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/273e0a10-0ba6-41bb-b8db-1ff672905957-istio-podinfo\") pod \"273e0a10-0ba6-41bb-b8db-1ff672905957\" (UID: \"273e0a10-0ba6-41bb-b8db-1ff672905957\") " Nov 02 01:44:17.024908 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:17.024457 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/273e0a10-0ba6-41bb-b8db-1ff672905957-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/273e0a10-0ba6-41bb-b8db-1ff672905957-istio-envoy\") pod \"273e0a10-0ba6-41bb-b8db-1ff672905957\" (UID: \"273e0a10-0ba6-41bb-b8db-1ff672905957\") " Nov 02 01:44:17.024908 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:17.024484 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/273e0a10-0ba6-41bb-b8db-1ff672905957-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/273e0a10-0ba6-41bb-b8db-1ff672905957-istio-data\") pod \"273e0a10-0ba6-41bb-b8db-1ff672905957\" (UID: \"273e0a10-0ba6-41bb-b8db-1ff672905957\") " Nov 02 01:44:17.024908 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:17.024511 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/273e0a10-0ba6-41bb-b8db-1ff672905957-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/273e0a10-0ba6-41bb-b8db-1ff672905957-credential-socket\") pod \"273e0a10-0ba6-41bb-b8db-1ff672905957\" (UID: \"273e0a10-0ba6-41bb-b8db-1ff672905957\") " Nov 02 01:44:17.024908 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:17.024545 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/273e0a10-0ba6-41bb-b8db-1ff672905957-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/273e0a10-0ba6-41bb-b8db-1ff672905957-workload-socket\") pod \"273e0a10-0ba6-41bb-b8db-1ff672905957\" (UID: \"273e0a10-0ba6-41bb-b8db-1ff672905957\") " Nov 02 01:44:17.024908 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:17.024620 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/273e0a10-0ba6-41bb-b8db-1ff672905957-istio-token\" (UniqueName: \"kubernetes.io/projected/273e0a10-0ba6-41bb-b8db-1ff672905957-istio-token\") pod \"273e0a10-0ba6-41bb-b8db-1ff672905957\" (UID: \"273e0a10-0ba6-41bb-b8db-1ff672905957\") " Nov 02 01:44:17.024908 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:17.024678 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/273e0a10-0ba6-41bb-b8db-1ff672905957-kube-api-access-zkc7r\" (UniqueName: \"kubernetes.io/projected/273e0a10-0ba6-41bb-b8db-1ff672905957-kube-api-access-zkc7r\") pod \"273e0a10-0ba6-41bb-b8db-1ff672905957\" (UID: \"273e0a10-0ba6-41bb-b8db-1ff672905957\") " Nov 02 01:44:17.041503 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:17.040769 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/273e0a10-0ba6-41bb-b8db-1ff672905957-istio-data" pod "273e0a10-0ba6-41bb-b8db-1ff672905957" (UID: "273e0a10-0ba6-41bb-b8db-1ff672905957"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:17.041503 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:17.040783 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/273e0a10-0ba6-41bb-b8db-1ff672905957-workload-socket" pod "273e0a10-0ba6-41bb-b8db-1ff672905957" (UID: "273e0a10-0ba6-41bb-b8db-1ff672905957"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:17.041503 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:17.040940 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/273e0a10-0ba6-41bb-b8db-1ff672905957-credential-socket" pod "273e0a10-0ba6-41bb-b8db-1ff672905957" (UID: "273e0a10-0ba6-41bb-b8db-1ff672905957"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:17.041503 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:17.041106 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/273e0a10-0ba6-41bb-b8db-1ff672905957-workload-certs" pod "273e0a10-0ba6-41bb-b8db-1ff672905957" (UID: "273e0a10-0ba6-41bb-b8db-1ff672905957"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:17.042328 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:17.041882 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/273e0a10-0ba6-41bb-b8db-1ff672905957-istiod-ca-cert" pod "273e0a10-0ba6-41bb-b8db-1ff672905957" (UID: "273e0a10-0ba6-41bb-b8db-1ff672905957"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:44:17.066909 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:17.064740 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/273e0a10-0ba6-41bb-b8db-1ff672905957-istio-envoy" pod "273e0a10-0ba6-41bb-b8db-1ff672905957" (UID: "273e0a10-0ba6-41bb-b8db-1ff672905957"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:17.069755 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:17.069690 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/273e0a10-0ba6-41bb-b8db-1ff672905957-kube-api-access-zkc7r" pod "273e0a10-0ba6-41bb-b8db-1ff672905957" (UID: "273e0a10-0ba6-41bb-b8db-1ff672905957"). InnerVolumeSpecName "kube-api-access-zkc7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:44:17.070924 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:17.070894 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/273e0a10-0ba6-41bb-b8db-1ff672905957-istio-token" pod "273e0a10-0ba6-41bb-b8db-1ff672905957" (UID: "273e0a10-0ba6-41bb-b8db-1ff672905957"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:44:17.072604 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:17.072576 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/273e0a10-0ba6-41bb-b8db-1ff672905957-istio-podinfo" pod "273e0a10-0ba6-41bb-b8db-1ff672905957" (UID: "273e0a10-0ba6-41bb-b8db-1ff672905957"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:44:17.134964 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:17.134926 4427 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/273e0a10-0ba6-41bb-b8db-1ff672905957-istiod-ca-cert\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:17.134964 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:17.134961 4427 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/273e0a10-0ba6-41bb-b8db-1ff672905957-istio-podinfo\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:17.134964 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:17.134971 4427 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/273e0a10-0ba6-41bb-b8db-1ff672905957-istio-envoy\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:17.135206 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:17.134979 4427 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/273e0a10-0ba6-41bb-b8db-1ff672905957-istio-data\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:17.135206 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:17.134989 4427 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/273e0a10-0ba6-41bb-b8db-1ff672905957-credential-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:17.135206 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:17.135029 4427 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/273e0a10-0ba6-41bb-b8db-1ff672905957-workload-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:17.135206 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:17.135038 4427 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/273e0a10-0ba6-41bb-b8db-1ff672905957-istio-token\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:17.135206 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:17.135047 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkc7r\" (UniqueName: \"kubernetes.io/projected/273e0a10-0ba6-41bb-b8db-1ff672905957-kube-api-access-zkc7r\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:17.135206 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:17.135057 4427 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/273e0a10-0ba6-41bb-b8db-1ff672905957-workload-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:17.536462 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:17.536428 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6da871e-dedf-412b-bbad-4257087b71a7" path="/var/lib/kubelet/pods/f6da871e-dedf-412b-bbad-4257087b71a7/volumes" Nov 02 01:44:17.798299 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:17.798225 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7" Nov 02 01:44:17.798484 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:17.798222 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7" event={"ID":"273e0a10-0ba6-41bb-b8db-1ff672905957","Type":"ContainerDied","Data":"c8a682cf493af60a41700bd561ddd67af9f5f2a0532b3d873a9e9470a83b415b"} Nov 02 01:44:17.798484 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:17.798355 4427 scope.go:122] "RemoveContainer" containerID="77854ebe0744657b26b6612e13f07e1bb749c776817ca86199013bbf26894742" Nov 02 01:44:17.841496 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:17.841461 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7"] Nov 02 01:44:17.843970 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:17.843947 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-9a4595-openshift-default-6449dbc487-sk2p7"] Nov 02 01:44:19.584910 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:19.584876 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="273e0a10-0ba6-41bb-b8db-1ff672905957" path="/var/lib/kubelet/pods/273e0a10-0ba6-41bb-b8db-1ff672905957/volumes" Nov 02 01:44:21.562606 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:21.562566 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9a4595-cc768d49f-fb6l8" podUID="fd3cc6d4-bb20-4c1d-a27f-b623408bafa4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 01:44:21.777125 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:21.777089 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b6ec07-d4f775449-gjvzd"] Nov 02 01:44:23.871805 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:23.871766 4427 generic.go:356] "Generic (PLEG): container finished" podID="fd3cc6d4-bb20-4c1d-a27f-b623408bafa4" containerID="d57cc647a5acf0d4dc1fdbfaf39d66e0cfc2d82b3a7982df3296a3cec576e614" exitCode=0 Nov 02 01:44:23.872206 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:23.871848 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9a4595-cc768d49f-fb6l8" event={"ID":"fd3cc6d4-bb20-4c1d-a27f-b623408bafa4","Type":"ContainerDied","Data":"d57cc647a5acf0d4dc1fdbfaf39d66e0cfc2d82b3a7982df3296a3cec576e614"} Nov 02 01:44:23.872206 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:23.871921 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9a4595-cc768d49f-fb6l8" event={"ID":"fd3cc6d4-bb20-4c1d-a27f-b623408bafa4","Type":"ContainerDied","Data":"de77d2a86c1970e6a33983c012564b2c8d7a2c26e5a0282a2f08fb87e01abacf"} Nov 02 01:44:23.872206 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:23.871947 4427 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de77d2a86c1970e6a33983c012564b2c8d7a2c26e5a0282a2f08fb87e01abacf" Nov 02 01:44:23.889874 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:23.889842 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9a4595-cc768d49f-fb6l8" Nov 02 01:44:23.985745 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:23.985709 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd3cc6d4-bb20-4c1d-a27f-b623408bafa4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd3cc6d4-bb20-4c1d-a27f-b623408bafa4-plugins-config-volume\") pod \"fd3cc6d4-bb20-4c1d-a27f-b623408bafa4\" (UID: \"fd3cc6d4-bb20-4c1d-a27f-b623408bafa4\") " Nov 02 01:44:23.985930 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:23.985837 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd3cc6d4-bb20-4c1d-a27f-b623408bafa4-kube-api-access-7j9m4\" (UniqueName: \"kubernetes.io/projected/fd3cc6d4-bb20-4c1d-a27f-b623408bafa4-kube-api-access-7j9m4\") pod \"fd3cc6d4-bb20-4c1d-a27f-b623408bafa4\" (UID: \"fd3cc6d4-bb20-4c1d-a27f-b623408bafa4\") " Nov 02 01:44:23.986168 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:23.986137 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd3cc6d4-bb20-4c1d-a27f-b623408bafa4-plugins-config-volume" pod "fd3cc6d4-bb20-4c1d-a27f-b623408bafa4" (UID: "fd3cc6d4-bb20-4c1d-a27f-b623408bafa4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:44:23.986627 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:23.986606 4427 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd3cc6d4-bb20-4c1d-a27f-b623408bafa4-plugins-config-volume\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:23.988255 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:23.988227 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd3cc6d4-bb20-4c1d-a27f-b623408bafa4-kube-api-access-7j9m4" pod "fd3cc6d4-bb20-4c1d-a27f-b623408bafa4" (UID: "fd3cc6d4-bb20-4c1d-a27f-b623408bafa4"). InnerVolumeSpecName "kube-api-access-7j9m4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:44:24.088851 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:24.088811 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j9m4\" (UniqueName: \"kubernetes.io/projected/fd3cc6d4-bb20-4c1d-a27f-b623408bafa4-kube-api-access-7j9m4\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:24.882138 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:24.882096 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9a4595-cc768d49f-fb6l8" Nov 02 01:44:24.912425 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:24.912365 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9a4595-cc768d49f-fb6l8"] Nov 02 01:44:24.916403 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:24.916358 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9a4595-cc768d49f-fb6l8"] Nov 02 01:44:25.556068 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:25.556034 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd3cc6d4-bb20-4c1d-a27f-b623408bafa4" path="/var/lib/kubelet/pods/fd3cc6d4-bb20-4c1d-a27f-b623408bafa4/volumes" Nov 02 01:44:25.556467 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:25.556451 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s"] Nov 02 01:44:25.556537 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:25.556486 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj"] Nov 02 01:44:25.556747 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:25.556715 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s" podUID="9a7a8380-cdbe-4204-a374-8d42f30b87d7" containerName="istio-proxy" containerID="cri-o://077360db8c1c94db2557c7917d6924ddd76d531b22434a58f7584e40ad8c79c7" gracePeriod=30 Nov 02 01:44:25.556916 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:25.556865 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj" podUID="eec37220-0f11-4a42-847f-363344172167" containerName="istio-proxy" containerID="cri-o://969856304df23df7a3ca8ac178919007726c6bb6c89de29257e3ea52d027456f" gracePeriod=30 Nov 02 01:44:26.777718 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:26.777675 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-b6ec07-d4f775449-gjvzd" podUID="7ad049fe-74e6-4083-92ed-09db1fb92ada" containerName="payload-processing" containerID="cri-o://d3363e7b99cf7a6a00ff13e395b4cf3a8c0c978d0dd1e825bf57d06593382b88" gracePeriod=25 Nov 02 01:44:28.283054 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:28.283013 4427 patch_prober.go:29] interesting pod/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 01:44:28.283539 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:28.283074 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj" podUID="eec37220-0f11-4a42-847f-363344172167" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 01:44:28.660212 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:28.660170 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl"] Nov 02 01:44:28.660428 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:28.660394 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl" podUID="00825c33-7e34-4522-adf8-142b8ae3ff73" containerName="istio-proxy" containerID="cri-o://4500bdb77b7beb118e1cecdef9ee744975e5d25d66798934008922007f4ec916" gracePeriod=30 Nov 02 01:44:30.869237 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:30.868975 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s" Nov 02 01:44:30.926095 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:30.926041 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj" Nov 02 01:44:30.939198 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:30.939165 4427 generic.go:356] "Generic (PLEG): container finished" podID="eec37220-0f11-4a42-847f-363344172167" containerID="969856304df23df7a3ca8ac178919007726c6bb6c89de29257e3ea52d027456f" exitCode=0 Nov 02 01:44:30.939431 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:30.939236 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj" Nov 02 01:44:30.939431 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:30.939251 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj" event={"ID":"eec37220-0f11-4a42-847f-363344172167","Type":"ContainerDied","Data":"969856304df23df7a3ca8ac178919007726c6bb6c89de29257e3ea52d027456f"} Nov 02 01:44:30.939431 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:30.939308 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj" event={"ID":"eec37220-0f11-4a42-847f-363344172167","Type":"ContainerDied","Data":"1de3d83b70990084b27410b5075685ad6b6b9217f81497ef9037c4052a27a7ed"} Nov 02 01:44:30.939431 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:30.939341 4427 scope.go:122] "RemoveContainer" containerID="969856304df23df7a3ca8ac178919007726c6bb6c89de29257e3ea52d027456f" Nov 02 01:44:30.941404 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:30.941362 4427 generic.go:356] "Generic (PLEG): container finished" podID="9a7a8380-cdbe-4204-a374-8d42f30b87d7" containerID="077360db8c1c94db2557c7917d6924ddd76d531b22434a58f7584e40ad8c79c7" exitCode=0 Nov 02 01:44:30.941510 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:30.941413 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s" event={"ID":"9a7a8380-cdbe-4204-a374-8d42f30b87d7","Type":"ContainerDied","Data":"077360db8c1c94db2557c7917d6924ddd76d531b22434a58f7584e40ad8c79c7"} Nov 02 01:44:30.941510 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:30.941474 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s" Nov 02 01:44:30.941510 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:30.941488 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s" event={"ID":"9a7a8380-cdbe-4204-a374-8d42f30b87d7","Type":"ContainerDied","Data":"40570b5065a75d0fc0429f07f80acceb1f98021703409e439fe451fa5b1c8955"} Nov 02 01:44:30.957175 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:30.956973 4427 scope.go:122] "RemoveContainer" containerID="969856304df23df7a3ca8ac178919007726c6bb6c89de29257e3ea52d027456f" Nov 02 01:44:30.957826 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:44:30.957791 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"969856304df23df7a3ca8ac178919007726c6bb6c89de29257e3ea52d027456f\": container with ID starting with 969856304df23df7a3ca8ac178919007726c6bb6c89de29257e3ea52d027456f not found: ID does not exist" containerID="969856304df23df7a3ca8ac178919007726c6bb6c89de29257e3ea52d027456f" Nov 02 01:44:30.957955 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:30.957836 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"969856304df23df7a3ca8ac178919007726c6bb6c89de29257e3ea52d027456f"} err="failed to get container status \"969856304df23df7a3ca8ac178919007726c6bb6c89de29257e3ea52d027456f\": rpc error: code = NotFound desc = could not find container \"969856304df23df7a3ca8ac178919007726c6bb6c89de29257e3ea52d027456f\": container with ID starting with 969856304df23df7a3ca8ac178919007726c6bb6c89de29257e3ea52d027456f not found: ID does not exist" Nov 02 01:44:30.957955 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:30.957862 4427 scope.go:122] "RemoveContainer" containerID="077360db8c1c94db2557c7917d6924ddd76d531b22434a58f7584e40ad8c79c7" Nov 02 01:44:30.967614 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:30.967581 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a7a8380-cdbe-4204-a374-8d42f30b87d7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a7a8380-cdbe-4204-a374-8d42f30b87d7-istio-podinfo\") pod \"9a7a8380-cdbe-4204-a374-8d42f30b87d7\" (UID: \"9a7a8380-cdbe-4204-a374-8d42f30b87d7\") " Nov 02 01:44:30.967788 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:30.967632 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a7a8380-cdbe-4204-a374-8d42f30b87d7-kube-api-access-gkqnd\" (UniqueName: \"kubernetes.io/projected/9a7a8380-cdbe-4204-a374-8d42f30b87d7-kube-api-access-gkqnd\") pod \"9a7a8380-cdbe-4204-a374-8d42f30b87d7\" (UID: \"9a7a8380-cdbe-4204-a374-8d42f30b87d7\") " Nov 02 01:44:30.967788 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:30.967665 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a7a8380-cdbe-4204-a374-8d42f30b87d7-istio-token\" (UniqueName: \"kubernetes.io/projected/9a7a8380-cdbe-4204-a374-8d42f30b87d7-istio-token\") pod \"9a7a8380-cdbe-4204-a374-8d42f30b87d7\" (UID: \"9a7a8380-cdbe-4204-a374-8d42f30b87d7\") " Nov 02 01:44:30.967933 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:30.967814 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a7a8380-cdbe-4204-a374-8d42f30b87d7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7a8380-cdbe-4204-a374-8d42f30b87d7-workload-socket\") pod \"9a7a8380-cdbe-4204-a374-8d42f30b87d7\" (UID: \"9a7a8380-cdbe-4204-a374-8d42f30b87d7\") " Nov 02 01:44:30.967995 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:30.967948 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a7a8380-cdbe-4204-a374-8d42f30b87d7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a7a8380-cdbe-4204-a374-8d42f30b87d7-istio-envoy\") pod \"9a7a8380-cdbe-4204-a374-8d42f30b87d7\" (UID: \"9a7a8380-cdbe-4204-a374-8d42f30b87d7\") " Nov 02 01:44:30.968057 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:30.968005 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a7a8380-cdbe-4204-a374-8d42f30b87d7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a7a8380-cdbe-4204-a374-8d42f30b87d7-workload-certs\") pod \"9a7a8380-cdbe-4204-a374-8d42f30b87d7\" (UID: \"9a7a8380-cdbe-4204-a374-8d42f30b87d7\") " Nov 02 01:44:30.968116 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:30.968072 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a7a8380-cdbe-4204-a374-8d42f30b87d7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a7a8380-cdbe-4204-a374-8d42f30b87d7-istio-data\") pod \"9a7a8380-cdbe-4204-a374-8d42f30b87d7\" (UID: \"9a7a8380-cdbe-4204-a374-8d42f30b87d7\") " Nov 02 01:44:30.968601 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:30.968211 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a7a8380-cdbe-4204-a374-8d42f30b87d7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7a8380-cdbe-4204-a374-8d42f30b87d7-credential-socket\") pod \"9a7a8380-cdbe-4204-a374-8d42f30b87d7\" (UID: \"9a7a8380-cdbe-4204-a374-8d42f30b87d7\") " Nov 02 01:44:30.968601 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:30.968272 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a7a8380-cdbe-4204-a374-8d42f30b87d7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a7a8380-cdbe-4204-a374-8d42f30b87d7-istiod-ca-cert\") pod \"9a7a8380-cdbe-4204-a374-8d42f30b87d7\" (UID: \"9a7a8380-cdbe-4204-a374-8d42f30b87d7\") " Nov 02 01:44:30.969486 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:30.969456 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a7a8380-cdbe-4204-a374-8d42f30b87d7-istiod-ca-cert" pod "9a7a8380-cdbe-4204-a374-8d42f30b87d7" (UID: "9a7a8380-cdbe-4204-a374-8d42f30b87d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:44:30.969746 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:30.969720 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a7a8380-cdbe-4204-a374-8d42f30b87d7-workload-socket" pod "9a7a8380-cdbe-4204-a374-8d42f30b87d7" (UID: "9a7a8380-cdbe-4204-a374-8d42f30b87d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:30.970285 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:30.970257 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a7a8380-cdbe-4204-a374-8d42f30b87d7-istio-data" pod "9a7a8380-cdbe-4204-a374-8d42f30b87d7" (UID: "9a7a8380-cdbe-4204-a374-8d42f30b87d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:30.970518 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:30.970483 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a7a8380-cdbe-4204-a374-8d42f30b87d7-workload-certs" pod "9a7a8380-cdbe-4204-a374-8d42f30b87d7" (UID: "9a7a8380-cdbe-4204-a374-8d42f30b87d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:30.970678 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:30.970666 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a7a8380-cdbe-4204-a374-8d42f30b87d7-credential-socket" pod "9a7a8380-cdbe-4204-a374-8d42f30b87d7" (UID: "9a7a8380-cdbe-4204-a374-8d42f30b87d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:30.971982 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:30.971953 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a7a8380-cdbe-4204-a374-8d42f30b87d7-istio-podinfo" pod "9a7a8380-cdbe-4204-a374-8d42f30b87d7" (UID: "9a7a8380-cdbe-4204-a374-8d42f30b87d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:44:30.974690 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:30.973884 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a7a8380-cdbe-4204-a374-8d42f30b87d7-istio-envoy" pod "9a7a8380-cdbe-4204-a374-8d42f30b87d7" (UID: "9a7a8380-cdbe-4204-a374-8d42f30b87d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:30.979071 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:30.979042 4427 scope.go:122] "RemoveContainer" containerID="077360db8c1c94db2557c7917d6924ddd76d531b22434a58f7584e40ad8c79c7" Nov 02 01:44:30.979480 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:44:30.979459 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"077360db8c1c94db2557c7917d6924ddd76d531b22434a58f7584e40ad8c79c7\": container with ID starting with 077360db8c1c94db2557c7917d6924ddd76d531b22434a58f7584e40ad8c79c7 not found: ID does not exist" containerID="077360db8c1c94db2557c7917d6924ddd76d531b22434a58f7584e40ad8c79c7" Nov 02 01:44:30.979595 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:30.979487 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"077360db8c1c94db2557c7917d6924ddd76d531b22434a58f7584e40ad8c79c7"} err="failed to get container status \"077360db8c1c94db2557c7917d6924ddd76d531b22434a58f7584e40ad8c79c7\": rpc error: code = NotFound desc = could not find container \"077360db8c1c94db2557c7917d6924ddd76d531b22434a58f7584e40ad8c79c7\": container with ID starting with 077360db8c1c94db2557c7917d6924ddd76d531b22434a58f7584e40ad8c79c7 not found: ID does not exist" Nov 02 01:44:30.979757 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:30.979740 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a7a8380-cdbe-4204-a374-8d42f30b87d7-istio-token" pod "9a7a8380-cdbe-4204-a374-8d42f30b87d7" (UID: "9a7a8380-cdbe-4204-a374-8d42f30b87d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:44:30.980038 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:30.980016 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a7a8380-cdbe-4204-a374-8d42f30b87d7-kube-api-access-gkqnd" pod "9a7a8380-cdbe-4204-a374-8d42f30b87d7" (UID: "9a7a8380-cdbe-4204-a374-8d42f30b87d7"). InnerVolumeSpecName "kube-api-access-gkqnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:44:31.080448 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.076918 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eec37220-0f11-4a42-847f-363344172167-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eec37220-0f11-4a42-847f-363344172167-workload-socket\") pod \"eec37220-0f11-4a42-847f-363344172167\" (UID: \"eec37220-0f11-4a42-847f-363344172167\") " Nov 02 01:44:31.080448 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.077037 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eec37220-0f11-4a42-847f-363344172167-kube-api-access-7n65v\" (UniqueName: \"kubernetes.io/projected/eec37220-0f11-4a42-847f-363344172167-kube-api-access-7n65v\") pod \"eec37220-0f11-4a42-847f-363344172167\" (UID: \"eec37220-0f11-4a42-847f-363344172167\") " Nov 02 01:44:31.082032 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.081570 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eec37220-0f11-4a42-847f-363344172167-workload-socket" pod "eec37220-0f11-4a42-847f-363344172167" (UID: "eec37220-0f11-4a42-847f-363344172167"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:31.087668 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.087618 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eec37220-0f11-4a42-847f-363344172167-istiod-ca-cert" pod "eec37220-0f11-4a42-847f-363344172167" (UID: "eec37220-0f11-4a42-847f-363344172167"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:44:31.113423 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.077113 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eec37220-0f11-4a42-847f-363344172167-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eec37220-0f11-4a42-847f-363344172167-istiod-ca-cert\") pod \"eec37220-0f11-4a42-847f-363344172167\" (UID: \"eec37220-0f11-4a42-847f-363344172167\") " Nov 02 01:44:31.113423 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.097518 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eec37220-0f11-4a42-847f-363344172167-istio-token\" (UniqueName: \"kubernetes.io/projected/eec37220-0f11-4a42-847f-363344172167-istio-token\") pod \"eec37220-0f11-4a42-847f-363344172167\" (UID: \"eec37220-0f11-4a42-847f-363344172167\") " Nov 02 01:44:31.113423 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.097610 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eec37220-0f11-4a42-847f-363344172167-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eec37220-0f11-4a42-847f-363344172167-workload-certs\") pod \"eec37220-0f11-4a42-847f-363344172167\" (UID: \"eec37220-0f11-4a42-847f-363344172167\") " Nov 02 01:44:31.113423 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.097659 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eec37220-0f11-4a42-847f-363344172167-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eec37220-0f11-4a42-847f-363344172167-istio-envoy\") pod \"eec37220-0f11-4a42-847f-363344172167\" (UID: \"eec37220-0f11-4a42-847f-363344172167\") " Nov 02 01:44:31.113423 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.097876 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eec37220-0f11-4a42-847f-363344172167-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eec37220-0f11-4a42-847f-363344172167-istio-podinfo\") pod \"eec37220-0f11-4a42-847f-363344172167\" (UID: \"eec37220-0f11-4a42-847f-363344172167\") " Nov 02 01:44:31.113423 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.097940 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eec37220-0f11-4a42-847f-363344172167-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eec37220-0f11-4a42-847f-363344172167-credential-socket\") pod \"eec37220-0f11-4a42-847f-363344172167\" (UID: \"eec37220-0f11-4a42-847f-363344172167\") " Nov 02 01:44:31.113423 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.097980 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eec37220-0f11-4a42-847f-363344172167-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eec37220-0f11-4a42-847f-363344172167-istio-data\") pod \"eec37220-0f11-4a42-847f-363344172167\" (UID: \"eec37220-0f11-4a42-847f-363344172167\") " Nov 02 01:44:31.113423 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.098878 4427 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a7a8380-cdbe-4204-a374-8d42f30b87d7-istio-data\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:31.113423 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.098904 4427 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eec37220-0f11-4a42-847f-363344172167-istiod-ca-cert\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:31.113423 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.098920 4427 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7a8380-cdbe-4204-a374-8d42f30b87d7-credential-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:31.113423 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.098949 4427 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a7a8380-cdbe-4204-a374-8d42f30b87d7-istiod-ca-cert\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:31.113423 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.098966 4427 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a7a8380-cdbe-4204-a374-8d42f30b87d7-istio-podinfo\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:31.113423 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.098982 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkqnd\" (UniqueName: \"kubernetes.io/projected/9a7a8380-cdbe-4204-a374-8d42f30b87d7-kube-api-access-gkqnd\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:31.113423 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.099008 4427 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a7a8380-cdbe-4204-a374-8d42f30b87d7-istio-token\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:31.113423 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.099022 4427 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7a8380-cdbe-4204-a374-8d42f30b87d7-workload-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:31.113423 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.099037 4427 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a7a8380-cdbe-4204-a374-8d42f30b87d7-istio-envoy\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:31.113423 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.099050 4427 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eec37220-0f11-4a42-847f-363344172167-workload-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:31.113423 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.099063 4427 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a7a8380-cdbe-4204-a374-8d42f30b87d7-workload-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:31.113423 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.105440 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eec37220-0f11-4a42-847f-363344172167-credential-socket" pod "eec37220-0f11-4a42-847f-363344172167" (UID: "eec37220-0f11-4a42-847f-363344172167"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:31.113423 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.105807 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eec37220-0f11-4a42-847f-363344172167-workload-certs" pod "eec37220-0f11-4a42-847f-363344172167" (UID: "eec37220-0f11-4a42-847f-363344172167"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:31.119742 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.119677 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eec37220-0f11-4a42-847f-363344172167-kube-api-access-7n65v" pod "eec37220-0f11-4a42-847f-363344172167" (UID: "eec37220-0f11-4a42-847f-363344172167"). InnerVolumeSpecName "kube-api-access-7n65v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:44:31.124307 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.120484 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eec37220-0f11-4a42-847f-363344172167-istio-data" pod "eec37220-0f11-4a42-847f-363344172167" (UID: "eec37220-0f11-4a42-847f-363344172167"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:31.127527 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.127498 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eec37220-0f11-4a42-847f-363344172167-istio-envoy" pod "eec37220-0f11-4a42-847f-363344172167" (UID: "eec37220-0f11-4a42-847f-363344172167"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:31.145726 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.145658 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eec37220-0f11-4a42-847f-363344172167-istio-token" pod "eec37220-0f11-4a42-847f-363344172167" (UID: "eec37220-0f11-4a42-847f-363344172167"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:44:31.147894 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.147864 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eec37220-0f11-4a42-847f-363344172167-istio-podinfo" pod "eec37220-0f11-4a42-847f-363344172167" (UID: "eec37220-0f11-4a42-847f-363344172167"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:44:31.201601 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.201557 4427 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eec37220-0f11-4a42-847f-363344172167-istio-podinfo\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:31.201874 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.201853 4427 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eec37220-0f11-4a42-847f-363344172167-credential-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:31.201992 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.201980 4427 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eec37220-0f11-4a42-847f-363344172167-istio-data\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:31.202094 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.202067 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n65v\" (UniqueName: \"kubernetes.io/projected/eec37220-0f11-4a42-847f-363344172167-kube-api-access-7n65v\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:31.202180 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.202169 4427 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eec37220-0f11-4a42-847f-363344172167-istio-token\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:31.202259 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.202248 4427 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eec37220-0f11-4a42-847f-363344172167-workload-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:31.202334 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.202323 4427 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eec37220-0f11-4a42-847f-363344172167-istio-envoy\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:31.290774 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.289642 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj"] Nov 02 01:44:31.299668 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.297752 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-xz7zj"] Nov 02 01:44:31.315881 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.315846 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s"] Nov 02 01:44:31.321260 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.321203 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-47dd19-openshift-default-6b95c479c9-gpp8s"] Nov 02 01:44:31.352841 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.352739 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" podUID="cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9" containerName="main" containerID="cri-o://6b6477df2cccdc50f6e7994f7b6adf0b04d30466c18186b06337a11b4401b868" gracePeriod=45 Nov 02 01:44:31.390084 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.390042 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" podUID="353a8a41-21ee-4470-861c-b53833beb106" containerName="main" containerID="cri-o://0901206f61d6087705a79fd16a87b60f61e8303a71c2cad4b8c40990a2f1a653" gracePeriod=45 Nov 02 01:44:31.553717 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.551812 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a7a8380-cdbe-4204-a374-8d42f30b87d7" path="/var/lib/kubelet/pods/9a7a8380-cdbe-4204-a374-8d42f30b87d7/volumes" Nov 02 01:44:31.553717 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.552785 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eec37220-0f11-4a42-847f-363344172167" path="/var/lib/kubelet/pods/eec37220-0f11-4a42-847f-363344172167/volumes" Nov 02 01:44:31.636132 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:44:31.636035 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:44:31.796748 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.796719 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" Nov 02 01:44:31.800148 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.800119 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" Nov 02 01:44:31.954051 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.953954 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/353a8a41-21ee-4470-861c-b53833beb106-kube-api-access-zm2p9\" (UniqueName: \"kubernetes.io/projected/353a8a41-21ee-4470-861c-b53833beb106-kube-api-access-zm2p9\") pod \"353a8a41-21ee-4470-861c-b53833beb106\" (UID: \"353a8a41-21ee-4470-861c-b53833beb106\") " Nov 02 01:44:31.954051 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.954044 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/353a8a41-21ee-4470-861c-b53833beb106-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/353a8a41-21ee-4470-861c-b53833beb106-tmp-dir\") pod \"353a8a41-21ee-4470-861c-b53833beb106\" (UID: \"353a8a41-21ee-4470-861c-b53833beb106\") " Nov 02 01:44:31.954667 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.954132 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9-dshm\") pod \"cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9\" (UID: \"cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9\") " Nov 02 01:44:31.954667 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.954197 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9-kube-api-access-rm5kp\" (UniqueName: \"kubernetes.io/projected/cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9-kube-api-access-rm5kp\") pod \"cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9\" (UID: \"cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9\") " Nov 02 01:44:31.954667 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.954244 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/353a8a41-21ee-4470-861c-b53833beb106-home\" (UniqueName: \"kubernetes.io/empty-dir/353a8a41-21ee-4470-861c-b53833beb106-home\") pod \"353a8a41-21ee-4470-861c-b53833beb106\" (UID: \"353a8a41-21ee-4470-861c-b53833beb106\") " Nov 02 01:44:31.954667 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.954276 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9-model-cache\") pod \"cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9\" (UID: \"cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9\") " Nov 02 01:44:31.954667 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.954315 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/353a8a41-21ee-4470-861c-b53833beb106-tls-certs\" (UniqueName: \"kubernetes.io/secret/353a8a41-21ee-4470-861c-b53833beb106-tls-certs\") pod \"353a8a41-21ee-4470-861c-b53833beb106\" (UID: \"353a8a41-21ee-4470-861c-b53833beb106\") " Nov 02 01:44:31.954667 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.954354 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/353a8a41-21ee-4470-861c-b53833beb106-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/353a8a41-21ee-4470-861c-b53833beb106-model-cache\") pod \"353a8a41-21ee-4470-861c-b53833beb106\" (UID: \"353a8a41-21ee-4470-861c-b53833beb106\") " Nov 02 01:44:31.954667 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.954483 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/353a8a41-21ee-4470-861c-b53833beb106-dshm\" (UniqueName: \"kubernetes.io/empty-dir/353a8a41-21ee-4470-861c-b53833beb106-dshm\") pod \"353a8a41-21ee-4470-861c-b53833beb106\" (UID: \"353a8a41-21ee-4470-861c-b53833beb106\") " Nov 02 01:44:31.954667 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.954558 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9-tmp-dir\") pod \"cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9\" (UID: \"cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9\") " Nov 02 01:44:31.955126 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.954642 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9-home\" (UniqueName: \"kubernetes.io/empty-dir/cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9-home\") pod \"cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9\" (UID: \"cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9\") " Nov 02 01:44:31.955126 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.954788 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9-tls-certs\" (UniqueName: \"kubernetes.io/secret/cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9-tls-certs\") pod \"cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9\" (UID: \"cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9\") " Nov 02 01:44:31.955638 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.955558 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/353a8a41-21ee-4470-861c-b53833beb106-tmp-dir" pod "353a8a41-21ee-4470-861c-b53833beb106" (UID: "353a8a41-21ee-4470-861c-b53833beb106"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:31.955638 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.955568 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/353a8a41-21ee-4470-861c-b53833beb106-home" pod "353a8a41-21ee-4470-861c-b53833beb106" (UID: "353a8a41-21ee-4470-861c-b53833beb106"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:31.955900 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.955870 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9-model-cache" pod "cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9" (UID: "cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:31.956263 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.956243 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9-tmp-dir" pod "cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9" (UID: "cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:31.957929 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.957591 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9-home" pod "cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9" (UID: "cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:31.958085 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.958060 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/353a8a41-21ee-4470-861c-b53833beb106-model-cache" pod "353a8a41-21ee-4470-861c-b53833beb106" (UID: "353a8a41-21ee-4470-861c-b53833beb106"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:31.959615 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.959592 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/353a8a41-21ee-4470-861c-b53833beb106-dshm" pod "353a8a41-21ee-4470-861c-b53833beb106" (UID: "353a8a41-21ee-4470-861c-b53833beb106"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:31.959740 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.959625 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9-dshm" pod "cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9" (UID: "cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:31.959740 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.959632 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/353a8a41-21ee-4470-861c-b53833beb106-kube-api-access-zm2p9" pod "353a8a41-21ee-4470-861c-b53833beb106" (UID: "353a8a41-21ee-4470-861c-b53833beb106"). InnerVolumeSpecName "kube-api-access-zm2p9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:44:31.960032 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.960009 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9-kube-api-access-rm5kp" pod "cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9" (UID: "cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9"). InnerVolumeSpecName "kube-api-access-rm5kp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:44:31.960456 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.960421 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9-tls-certs" pod "cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9" (UID: "cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:44:31.960597 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.960578 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/353a8a41-21ee-4470-861c-b53833beb106-tls-certs" pod "353a8a41-21ee-4470-861c-b53833beb106" (UID: "353a8a41-21ee-4470-861c-b53833beb106"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:44:31.961948 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.961922 4427 generic.go:356] "Generic (PLEG): container finished" podID="cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9" containerID="6b6477df2cccdc50f6e7994f7b6adf0b04d30466c18186b06337a11b4401b868" exitCode=0 Nov 02 01:44:31.962058 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.962039 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" Nov 02 01:44:31.962129 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.962057 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" event={"ID":"cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9","Type":"ContainerDied","Data":"6b6477df2cccdc50f6e7994f7b6adf0b04d30466c18186b06337a11b4401b868"} Nov 02 01:44:31.962129 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.962114 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" event={"ID":"cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9","Type":"ContainerDied","Data":"0e4d0b87c85e2de7e0a1d7b49a6e8709e9ffd52c251cae0cca36c868fa3c298d"} Nov 02 01:44:31.962244 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.962139 4427 scope.go:122] "RemoveContainer" containerID="6b6477df2cccdc50f6e7994f7b6adf0b04d30466c18186b06337a11b4401b868" Nov 02 01:44:31.967570 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.967540 4427 generic.go:356] "Generic (PLEG): container finished" podID="353a8a41-21ee-4470-861c-b53833beb106" containerID="0901206f61d6087705a79fd16a87b60f61e8303a71c2cad4b8c40990a2f1a653" exitCode=0 Nov 02 01:44:31.967687 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.967594 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" Nov 02 01:44:31.967687 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.967680 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" event={"ID":"353a8a41-21ee-4470-861c-b53833beb106","Type":"ContainerDied","Data":"0901206f61d6087705a79fd16a87b60f61e8303a71c2cad4b8c40990a2f1a653"} Nov 02 01:44:31.967799 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.967712 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7" event={"ID":"353a8a41-21ee-4470-861c-b53833beb106","Type":"ContainerDied","Data":"8af4b67ddc044e34300cb434ec421ebb8060a5136a1b8f478421b3defd7f9167"} Nov 02 01:44:31.990038 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.990014 4427 scope.go:122] "RemoveContainer" containerID="6b6477df2cccdc50f6e7994f7b6adf0b04d30466c18186b06337a11b4401b868" Nov 02 01:44:31.990496 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:44:31.990461 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b6477df2cccdc50f6e7994f7b6adf0b04d30466c18186b06337a11b4401b868\": container with ID starting with 6b6477df2cccdc50f6e7994f7b6adf0b04d30466c18186b06337a11b4401b868 not found: ID does not exist" containerID="6b6477df2cccdc50f6e7994f7b6adf0b04d30466c18186b06337a11b4401b868" Nov 02 01:44:31.990653 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.990511 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b6477df2cccdc50f6e7994f7b6adf0b04d30466c18186b06337a11b4401b868"} err="failed to get container status \"6b6477df2cccdc50f6e7994f7b6adf0b04d30466c18186b06337a11b4401b868\": rpc error: code = NotFound desc = could not find container \"6b6477df2cccdc50f6e7994f7b6adf0b04d30466c18186b06337a11b4401b868\": container with ID starting with 6b6477df2cccdc50f6e7994f7b6adf0b04d30466c18186b06337a11b4401b868 not found: ID does not exist" Nov 02 01:44:31.990653 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:31.990541 4427 scope.go:122] "RemoveContainer" containerID="0901206f61d6087705a79fd16a87b60f61e8303a71c2cad4b8c40990a2f1a653" Nov 02 01:44:32.002012 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:32.001992 4427 scope.go:122] "RemoveContainer" containerID="0901206f61d6087705a79fd16a87b60f61e8303a71c2cad4b8c40990a2f1a653" Nov 02 01:44:32.002367 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:44:32.002342 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0901206f61d6087705a79fd16a87b60f61e8303a71c2cad4b8c40990a2f1a653\": container with ID starting with 0901206f61d6087705a79fd16a87b60f61e8303a71c2cad4b8c40990a2f1a653 not found: ID does not exist" containerID="0901206f61d6087705a79fd16a87b60f61e8303a71c2cad4b8c40990a2f1a653" Nov 02 01:44:32.002480 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:32.002419 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0901206f61d6087705a79fd16a87b60f61e8303a71c2cad4b8c40990a2f1a653"} err="failed to get container status \"0901206f61d6087705a79fd16a87b60f61e8303a71c2cad4b8c40990a2f1a653\": rpc error: code = NotFound desc = could not find container \"0901206f61d6087705a79fd16a87b60f61e8303a71c2cad4b8c40990a2f1a653\": container with ID starting with 0901206f61d6087705a79fd16a87b60f61e8303a71c2cad4b8c40990a2f1a653 not found: ID does not exist" Nov 02 01:44:32.003979 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:32.003956 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv"] Nov 02 01:44:32.008249 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:32.008227 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv"] Nov 02 01:44:32.019191 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:32.019164 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7"] Nov 02 01:44:32.025150 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:32.025128 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-b6ec07/distinct-w4-b6ec07-kserve-fc47dc9b4-wfcf7"] Nov 02 01:44:32.057313 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:32.057284 4427 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9-tmp-dir\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:32.057313 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:32.057310 4427 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9-home\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:32.057313 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:32.057319 4427 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9-tls-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:32.057541 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:32.057330 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm2p9\" (UniqueName: \"kubernetes.io/projected/353a8a41-21ee-4470-861c-b53833beb106-kube-api-access-zm2p9\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:32.057541 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:32.057342 4427 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/353a8a41-21ee-4470-861c-b53833beb106-tmp-dir\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:32.057541 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:32.057350 4427 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9-dshm\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:32.057541 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:32.057358 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm5kp\" (UniqueName: \"kubernetes.io/projected/cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9-kube-api-access-rm5kp\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:32.057541 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:32.057369 4427 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/353a8a41-21ee-4470-861c-b53833beb106-home\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:32.057541 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:32.057413 4427 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9-model-cache\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:32.057541 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:32.057422 4427 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/353a8a41-21ee-4470-861c-b53833beb106-tls-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:32.057541 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:32.057430 4427 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/353a8a41-21ee-4470-861c-b53833beb106-model-cache\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:32.057541 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:32.057438 4427 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/353a8a41-21ee-4470-861c-b53833beb106-dshm\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:32.751962 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:32.751917 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b6ec07/distinct-2-w4-b6ec07-kserve-6575469d47-l7crv" podUID="cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:44:33.245322 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:33.245275 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-b6ec07-d4f775449-gjvzd" podUID="7ad049fe-74e6-4083-92ed-09db1fb92ada" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 01:44:33.539273 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:33.539194 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="353a8a41-21ee-4470-861c-b53833beb106" path="/var/lib/kubelet/pods/353a8a41-21ee-4470-861c-b53833beb106/volumes" Nov 02 01:44:33.539762 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:33.539745 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9" path="/var/lib/kubelet/pods/cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9/volumes" Nov 02 01:44:33.990951 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:33.990919 4427 generic.go:356] "Generic (PLEG): container finished" podID="00825c33-7e34-4522-adf8-142b8ae3ff73" containerID="4500bdb77b7beb118e1cecdef9ee744975e5d25d66798934008922007f4ec916" exitCode=0 Nov 02 01:44:33.991175 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:33.991008 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl" event={"ID":"00825c33-7e34-4522-adf8-142b8ae3ff73","Type":"ContainerDied","Data":"4500bdb77b7beb118e1cecdef9ee744975e5d25d66798934008922007f4ec916"} Nov 02 01:44:33.991175 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:33.991060 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl" event={"ID":"00825c33-7e34-4522-adf8-142b8ae3ff73","Type":"ContainerDied","Data":"1271970a45475e3a6c770dc4b9dca769c159f2d00e9b4a4165839b11a6343945"} Nov 02 01:44:33.991175 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:33.991083 4427 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1271970a45475e3a6c770dc4b9dca769c159f2d00e9b4a4165839b11a6343945" Nov 02 01:44:33.995570 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:33.995549 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl" Nov 02 01:44:34.091836 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:34.091749 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00825c33-7e34-4522-adf8-142b8ae3ff73-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00825c33-7e34-4522-adf8-142b8ae3ff73-istio-envoy\") pod \"00825c33-7e34-4522-adf8-142b8ae3ff73\" (UID: \"00825c33-7e34-4522-adf8-142b8ae3ff73\") " Nov 02 01:44:34.092096 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:34.091840 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/00825c33-7e34-4522-adf8-142b8ae3ff73-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00825c33-7e34-4522-adf8-142b8ae3ff73-istio-podinfo\") pod \"00825c33-7e34-4522-adf8-142b8ae3ff73\" (UID: \"00825c33-7e34-4522-adf8-142b8ae3ff73\") " Nov 02 01:44:34.092096 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:34.091953 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00825c33-7e34-4522-adf8-142b8ae3ff73-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00825c33-7e34-4522-adf8-142b8ae3ff73-istiod-ca-cert\") pod \"00825c33-7e34-4522-adf8-142b8ae3ff73\" (UID: \"00825c33-7e34-4522-adf8-142b8ae3ff73\") " Nov 02 01:44:34.092096 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:34.092004 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00825c33-7e34-4522-adf8-142b8ae3ff73-istio-token\" (UniqueName: \"kubernetes.io/projected/00825c33-7e34-4522-adf8-142b8ae3ff73-istio-token\") pod \"00825c33-7e34-4522-adf8-142b8ae3ff73\" (UID: \"00825c33-7e34-4522-adf8-142b8ae3ff73\") " Nov 02 01:44:34.092278 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:34.092101 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00825c33-7e34-4522-adf8-142b8ae3ff73-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00825c33-7e34-4522-adf8-142b8ae3ff73-workload-certs\") pod \"00825c33-7e34-4522-adf8-142b8ae3ff73\" (UID: \"00825c33-7e34-4522-adf8-142b8ae3ff73\") " Nov 02 01:44:34.092278 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:34.092171 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00825c33-7e34-4522-adf8-142b8ae3ff73-kube-api-access-hxmd4\" (UniqueName: \"kubernetes.io/projected/00825c33-7e34-4522-adf8-142b8ae3ff73-kube-api-access-hxmd4\") pod \"00825c33-7e34-4522-adf8-142b8ae3ff73\" (UID: \"00825c33-7e34-4522-adf8-142b8ae3ff73\") " Nov 02 01:44:34.092451 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:34.092417 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00825c33-7e34-4522-adf8-142b8ae3ff73-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00825c33-7e34-4522-adf8-142b8ae3ff73-workload-socket\") pod \"00825c33-7e34-4522-adf8-142b8ae3ff73\" (UID: \"00825c33-7e34-4522-adf8-142b8ae3ff73\") " Nov 02 01:44:34.092522 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:34.092436 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00825c33-7e34-4522-adf8-142b8ae3ff73-istiod-ca-cert" pod "00825c33-7e34-4522-adf8-142b8ae3ff73" (UID: "00825c33-7e34-4522-adf8-142b8ae3ff73"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:44:34.092522 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:34.092457 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00825c33-7e34-4522-adf8-142b8ae3ff73-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00825c33-7e34-4522-adf8-142b8ae3ff73-istio-data\") pod \"00825c33-7e34-4522-adf8-142b8ae3ff73\" (UID: \"00825c33-7e34-4522-adf8-142b8ae3ff73\") " Nov 02 01:44:34.092652 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:34.092538 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00825c33-7e34-4522-adf8-142b8ae3ff73-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00825c33-7e34-4522-adf8-142b8ae3ff73-credential-socket\") pod \"00825c33-7e34-4522-adf8-142b8ae3ff73\" (UID: \"00825c33-7e34-4522-adf8-142b8ae3ff73\") " Nov 02 01:44:34.092778 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:34.092753 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00825c33-7e34-4522-adf8-142b8ae3ff73-workload-socket" pod "00825c33-7e34-4522-adf8-142b8ae3ff73" (UID: "00825c33-7e34-4522-adf8-142b8ae3ff73"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:34.093000 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:34.092972 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00825c33-7e34-4522-adf8-142b8ae3ff73-credential-socket" pod "00825c33-7e34-4522-adf8-142b8ae3ff73" (UID: "00825c33-7e34-4522-adf8-142b8ae3ff73"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:34.093095 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:34.092999 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00825c33-7e34-4522-adf8-142b8ae3ff73-istio-data" pod "00825c33-7e34-4522-adf8-142b8ae3ff73" (UID: "00825c33-7e34-4522-adf8-142b8ae3ff73"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:34.093447 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:34.093424 4427 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00825c33-7e34-4522-adf8-142b8ae3ff73-istiod-ca-cert\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:34.093708 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:34.093452 4427 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00825c33-7e34-4522-adf8-142b8ae3ff73-workload-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:34.093708 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:34.093467 4427 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00825c33-7e34-4522-adf8-142b8ae3ff73-istio-data\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:34.093708 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:34.093483 4427 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00825c33-7e34-4522-adf8-142b8ae3ff73-credential-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:34.093708 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:34.093446 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00825c33-7e34-4522-adf8-142b8ae3ff73-workload-certs" pod "00825c33-7e34-4522-adf8-142b8ae3ff73" (UID: "00825c33-7e34-4522-adf8-142b8ae3ff73"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:34.094743 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:34.094718 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00825c33-7e34-4522-adf8-142b8ae3ff73-istio-podinfo" pod "00825c33-7e34-4522-adf8-142b8ae3ff73" (UID: "00825c33-7e34-4522-adf8-142b8ae3ff73"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:44:34.094878 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:34.094856 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00825c33-7e34-4522-adf8-142b8ae3ff73-istio-envoy" pod "00825c33-7e34-4522-adf8-142b8ae3ff73" (UID: "00825c33-7e34-4522-adf8-142b8ae3ff73"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:34.094980 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:34.094956 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00825c33-7e34-4522-adf8-142b8ae3ff73-istio-token" pod "00825c33-7e34-4522-adf8-142b8ae3ff73" (UID: "00825c33-7e34-4522-adf8-142b8ae3ff73"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:44:34.096428 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:34.096391 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00825c33-7e34-4522-adf8-142b8ae3ff73-kube-api-access-hxmd4" pod "00825c33-7e34-4522-adf8-142b8ae3ff73" (UID: "00825c33-7e34-4522-adf8-142b8ae3ff73"). InnerVolumeSpecName "kube-api-access-hxmd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:44:34.195336 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:34.195297 4427 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00825c33-7e34-4522-adf8-142b8ae3ff73-istio-envoy\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:34.195336 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:34.195330 4427 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00825c33-7e34-4522-adf8-142b8ae3ff73-istio-podinfo\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:34.195336 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:34.195343 4427 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00825c33-7e34-4522-adf8-142b8ae3ff73-istio-token\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:34.195640 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:34.195357 4427 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00825c33-7e34-4522-adf8-142b8ae3ff73-workload-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:34.195640 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:34.195369 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxmd4\" (UniqueName: \"kubernetes.io/projected/00825c33-7e34-4522-adf8-142b8ae3ff73-kube-api-access-hxmd4\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:35.001988 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:35.001957 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl" Nov 02 01:44:35.037992 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:35.036406 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl"] Nov 02 01:44:35.042360 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:35.042329 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-00f1da-openshift-default-5fbc9cc4cf-ljpbl"] Nov 02 01:44:35.542396 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:35.542314 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00825c33-7e34-4522-adf8-142b8ae3ff73" path="/var/lib/kubelet/pods/00825c33-7e34-4522-adf8-142b8ae3ff73/volumes" Nov 02 01:44:36.340713 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.340676 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93a027b3-m87mq"] Nov 02 01:44:36.341333 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341314 4427 cpu_manager.go:414] "containerMap: removing container" podUID="ea61971c-5a72-4609-b5a0-58a74b324bb7" containerName="main" Nov 02 01:44:36.341333 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341333 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea61971c-5a72-4609-b5a0-58a74b324bb7" containerName="main" Nov 02 01:44:36.341555 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341347 4427 cpu_manager.go:414] "containerMap: removing container" podUID="f6da871e-dedf-412b-bbad-4257087b71a7" containerName="istio-proxy" Nov 02 01:44:36.341555 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341352 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6da871e-dedf-412b-bbad-4257087b71a7" containerName="istio-proxy" Nov 02 01:44:36.341555 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341359 4427 cpu_manager.go:414] "containerMap: removing container" podUID="57445e45-2af1-475b-a32b-e8d88a5cdc54" containerName="maas-api" Nov 02 01:44:36.341555 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341364 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57445e45-2af1-475b-a32b-e8d88a5cdc54" containerName="maas-api" Nov 02 01:44:36.341555 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341389 4427 cpu_manager.go:414] "containerMap: removing container" podUID="f6da871e-dedf-412b-bbad-4257087b71a7" containerName="istio-proxy" Nov 02 01:44:36.341555 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341397 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6da871e-dedf-412b-bbad-4257087b71a7" containerName="istio-proxy" Nov 02 01:44:36.341555 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341406 4427 cpu_manager.go:414] "containerMap: removing container" podUID="c227e450-ed88-4cb4-a5fd-e88f150d14ab" containerName="istio-proxy" Nov 02 01:44:36.341555 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341413 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c227e450-ed88-4cb4-a5fd-e88f150d14ab" containerName="istio-proxy" Nov 02 01:44:36.341555 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341426 4427 cpu_manager.go:414] "containerMap: removing container" podUID="00825c33-7e34-4522-adf8-142b8ae3ff73" containerName="istio-proxy" Nov 02 01:44:36.341555 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341432 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00825c33-7e34-4522-adf8-142b8ae3ff73" containerName="istio-proxy" Nov 02 01:44:36.341555 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341440 4427 cpu_manager.go:414] "containerMap: removing container" podUID="c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6" containerName="main" Nov 02 01:44:36.341555 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341446 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6" containerName="main" Nov 02 01:44:36.341555 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341454 4427 cpu_manager.go:414] "containerMap: removing container" podUID="353a8a41-21ee-4470-861c-b53833beb106" containerName="main" Nov 02 01:44:36.341555 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341462 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="353a8a41-21ee-4470-861c-b53833beb106" containerName="main" Nov 02 01:44:36.341555 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341475 4427 cpu_manager.go:414] "containerMap: removing container" podUID="b9bb7aa5-03de-42b9-a6d3-b1d8e7806a74" containerName="payload-processing" Nov 02 01:44:36.341555 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341482 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9bb7aa5-03de-42b9-a6d3-b1d8e7806a74" containerName="payload-processing" Nov 02 01:44:36.341555 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341491 4427 cpu_manager.go:414] "containerMap: removing container" podUID="b45e955d-0d37-44c2-96be-42dc39687913" containerName="main" Nov 02 01:44:36.341555 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341498 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b45e955d-0d37-44c2-96be-42dc39687913" containerName="main" Nov 02 01:44:36.341555 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341509 4427 cpu_manager.go:414] "containerMap: removing container" podUID="51776e18-2d31-454f-8c33-d7fbb169c601" containerName="istio-proxy" Nov 02 01:44:36.341555 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341515 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51776e18-2d31-454f-8c33-d7fbb169c601" containerName="istio-proxy" Nov 02 01:44:36.341555 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341530 4427 cpu_manager.go:414] "containerMap: removing container" podUID="e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0" containerName="istio-proxy" Nov 02 01:44:36.341555 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341537 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0" containerName="istio-proxy" Nov 02 01:44:36.341555 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341546 4427 cpu_manager.go:414] "containerMap: removing container" podUID="3d9b8d38-b3ce-4556-8580-93cfd36b2d2a" containerName="payload-pre-processing" Nov 02 01:44:36.341555 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341554 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d9b8d38-b3ce-4556-8580-93cfd36b2d2a" containerName="payload-pre-processing" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341575 4427 cpu_manager.go:414] "containerMap: removing container" podUID="9a7a8380-cdbe-4204-a374-8d42f30b87d7" containerName="istio-proxy" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341582 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a7a8380-cdbe-4204-a374-8d42f30b87d7" containerName="istio-proxy" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341602 4427 cpu_manager.go:414] "containerMap: removing container" podUID="c3d179f1-c7e0-4340-a286-2efbbc44ed53" containerName="istio-proxy" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341606 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3d179f1-c7e0-4340-a286-2efbbc44ed53" containerName="istio-proxy" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341619 4427 cpu_manager.go:414] "containerMap: removing container" podUID="0ed6780b-6074-4ce4-b9c0-44253b32955c" containerName="payload-pre-processing" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341626 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ed6780b-6074-4ce4-b9c0-44253b32955c" containerName="payload-pre-processing" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341632 4427 cpu_manager.go:414] "containerMap: removing container" podUID="d10ccd31-3d14-498e-bf7a-0b3034576528" containerName="payload-pre-processing" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341637 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d10ccd31-3d14-498e-bf7a-0b3034576528" containerName="payload-pre-processing" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341643 4427 cpu_manager.go:414] "containerMap: removing container" podUID="51776e18-2d31-454f-8c33-d7fbb169c601" containerName="istio-proxy" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341647 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51776e18-2d31-454f-8c33-d7fbb169c601" containerName="istio-proxy" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341660 4427 cpu_manager.go:414] "containerMap: removing container" podUID="eec37220-0f11-4a42-847f-363344172167" containerName="istio-proxy" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341664 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eec37220-0f11-4a42-847f-363344172167" containerName="istio-proxy" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341672 4427 cpu_manager.go:414] "containerMap: removing container" podUID="23707aeb-a006-4049-bca4-074bbd446ca7" containerName="payload-processing" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341676 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23707aeb-a006-4049-bca4-074bbd446ca7" containerName="payload-processing" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341682 4427 cpu_manager.go:414] "containerMap: removing container" podUID="fd3cc6d4-bb20-4c1d-a27f-b623408bafa4" containerName="payload-pre-processing" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341685 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd3cc6d4-bb20-4c1d-a27f-b623408bafa4" containerName="payload-pre-processing" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341696 4427 cpu_manager.go:414] "containerMap: removing container" podUID="a6fe33af-426d-4155-98a1-a6dfdaeead32" containerName="istio-proxy" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341701 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6fe33af-426d-4155-98a1-a6dfdaeead32" containerName="istio-proxy" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341708 4427 cpu_manager.go:414] "containerMap: removing container" podUID="c3d179f1-c7e0-4340-a286-2efbbc44ed53" containerName="istio-proxy" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341712 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3d179f1-c7e0-4340-a286-2efbbc44ed53" containerName="istio-proxy" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341718 4427 cpu_manager.go:414] "containerMap: removing container" podUID="1371b4cc-965f-4e8c-a482-6f2e4c76282e" containerName="istio-proxy" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341722 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1371b4cc-965f-4e8c-a482-6f2e4c76282e" containerName="istio-proxy" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341730 4427 cpu_manager.go:414] "containerMap: removing container" podUID="273e0a10-0ba6-41bb-b8db-1ff672905957" containerName="istio-proxy" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341734 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="273e0a10-0ba6-41bb-b8db-1ff672905957" containerName="istio-proxy" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341741 4427 cpu_manager.go:414] "containerMap: removing container" podUID="273e0a10-0ba6-41bb-b8db-1ff672905957" containerName="istio-proxy" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341745 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="273e0a10-0ba6-41bb-b8db-1ff672905957" containerName="istio-proxy" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341752 4427 cpu_manager.go:414] "containerMap: removing container" podUID="5ff70346-02a9-4a43-9b5f-19a83d64d169" containerName="payload-pre-processing" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341758 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ff70346-02a9-4a43-9b5f-19a83d64d169" containerName="payload-pre-processing" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341770 4427 cpu_manager.go:414] "containerMap: removing container" podUID="3869bb0d-5d24-4a09-bd83-9aea716bfd90" containerName="maas-api" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341777 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3869bb0d-5d24-4a09-bd83-9aea716bfd90" containerName="maas-api" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341790 4427 cpu_manager.go:414] "containerMap: removing container" podUID="53ae6505-044c-4e0d-8962-7aa6eb262ac1" containerName="payload-processing" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341797 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53ae6505-044c-4e0d-8962-7aa6eb262ac1" containerName="payload-processing" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341807 4427 cpu_manager.go:414] "containerMap: removing container" podUID="c227e450-ed88-4cb4-a5fd-e88f150d14ab" containerName="istio-proxy" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341812 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c227e450-ed88-4cb4-a5fd-e88f150d14ab" containerName="istio-proxy" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341822 4427 cpu_manager.go:414] "containerMap: removing container" podUID="8f3cd62e-2f33-481d-a827-63d2fddf6a46" containerName="payload-processing" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341826 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f3cd62e-2f33-481d-a827-63d2fddf6a46" containerName="payload-processing" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341833 4427 cpu_manager.go:414] "containerMap: removing container" podUID="1e58e874-db28-4a0e-8b46-368b6d4d0960" containerName="payload-processing" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341837 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e58e874-db28-4a0e-8b46-368b6d4d0960" containerName="payload-processing" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341843 4427 cpu_manager.go:414] "containerMap: removing container" podUID="a097a1f1-7f03-479d-a65a-90f8e542d06b" containerName="payload-pre-processing" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341847 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a097a1f1-7f03-479d-a65a-90f8e542d06b" containerName="payload-pre-processing" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341855 4427 cpu_manager.go:414] "containerMap: removing container" podUID="95975531-e9c3-4fcb-9f93-ddab47bb62ba" containerName="istio-proxy" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341859 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95975531-e9c3-4fcb-9f93-ddab47bb62ba" containerName="istio-proxy" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341865 4427 cpu_manager.go:414] "containerMap: removing container" podUID="e1412019-62bb-4fc3-a7ad-8e123fef90eb" containerName="istio-proxy" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341869 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1412019-62bb-4fc3-a7ad-8e123fef90eb" containerName="istio-proxy" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341876 4427 cpu_manager.go:414] "containerMap: removing container" podUID="6b8221ff-cb53-42c0-a697-4992c40d8e52" containerName="istio-proxy" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341880 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b8221ff-cb53-42c0-a697-4992c40d8e52" containerName="istio-proxy" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341890 4427 cpu_manager.go:414] "containerMap: removing container" podUID="cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9" containerName="main" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.341895 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9" containerName="main" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.342021 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1412019-62bb-4fc3-a7ad-8e123fef90eb" containerName="istio-proxy" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.342032 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b8221ff-cb53-42c0-a697-4992c40d8e52" containerName="istio-proxy" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.342041 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd5f5a8c-cac2-4b9b-868b-43f4d70ba5f9" containerName="main" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.342051 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea61971c-5a72-4609-b5a0-58a74b324bb7" containerName="main" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.342059 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6da871e-dedf-412b-bbad-4257087b71a7" containerName="istio-proxy" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.342067 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="57445e45-2af1-475b-a32b-e8d88a5cdc54" containerName="maas-api" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.342077 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6da871e-dedf-412b-bbad-4257087b71a7" containerName="istio-proxy" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.342087 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="c227e450-ed88-4cb4-a5fd-e88f150d14ab" containerName="istio-proxy" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.342096 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="00825c33-7e34-4522-adf8-142b8ae3ff73" containerName="istio-proxy" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.342104 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5dfd8e4-df57-4cfb-8ffd-d44dbb0273b6" containerName="main" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.342112 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="353a8a41-21ee-4470-861c-b53833beb106" containerName="main" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.342121 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9bb7aa5-03de-42b9-a6d3-b1d8e7806a74" containerName="payload-processing" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.342129 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="b45e955d-0d37-44c2-96be-42dc39687913" containerName="main" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.342139 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="51776e18-2d31-454f-8c33-d7fbb169c601" containerName="istio-proxy" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.342147 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0" containerName="istio-proxy" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.342155 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d9b8d38-b3ce-4556-8580-93cfd36b2d2a" containerName="payload-pre-processing" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.342164 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a7a8380-cdbe-4204-a374-8d42f30b87d7" containerName="istio-proxy" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.342175 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3d179f1-c7e0-4340-a286-2efbbc44ed53" containerName="istio-proxy" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.342182 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ed6780b-6074-4ce4-b9c0-44253b32955c" containerName="payload-pre-processing" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.342193 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="d10ccd31-3d14-498e-bf7a-0b3034576528" containerName="payload-pre-processing" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.342204 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="eec37220-0f11-4a42-847f-363344172167" containerName="istio-proxy" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.342212 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="23707aeb-a006-4049-bca4-074bbd446ca7" containerName="payload-processing" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.342219 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd3cc6d4-bb20-4c1d-a27f-b623408bafa4" containerName="payload-pre-processing" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.342229 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6fe33af-426d-4155-98a1-a6dfdaeead32" containerName="istio-proxy" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.342239 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3d179f1-c7e0-4340-a286-2efbbc44ed53" containerName="istio-proxy" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.342249 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="1371b4cc-965f-4e8c-a482-6f2e4c76282e" containerName="istio-proxy" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.342258 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="273e0a10-0ba6-41bb-b8db-1ff672905957" containerName="istio-proxy" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.342268 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="273e0a10-0ba6-41bb-b8db-1ff672905957" containerName="istio-proxy" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.342277 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ff70346-02a9-4a43-9b5f-19a83d64d169" containerName="payload-pre-processing" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.342285 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="3869bb0d-5d24-4a09-bd83-9aea716bfd90" containerName="maas-api" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.342294 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="53ae6505-044c-4e0d-8962-7aa6eb262ac1" containerName="payload-processing" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.342304 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f3cd62e-2f33-481d-a827-63d2fddf6a46" containerName="payload-processing" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.342311 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e58e874-db28-4a0e-8b46-368b6d4d0960" containerName="payload-processing" Nov 02 01:44:36.342441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.342319 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="a097a1f1-7f03-479d-a65a-90f8e542d06b" containerName="payload-pre-processing" Nov 02 01:44:36.349101 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.349082 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a027b3-m87mq" Nov 02 01:44:36.353616 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.353596 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 01:44:36.354216 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.354197 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\"" Nov 02 01:44:36.357579 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.357558 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93a027b3-m87mq"] Nov 02 01:44:36.582444 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.575245 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f20bf903-25ba-4be2-8c17-2a10a0255e13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93a027b3-m87mq\" (UID: \"f20bf903-25ba-4be2-8c17-2a10a0255e13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a027b3-m87mq" Nov 02 01:44:36.582444 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.575346 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpzfm\" (UniqueName: \"kubernetes.io/projected/f20bf903-25ba-4be2-8c17-2a10a0255e13-kube-api-access-tpzfm\") pod \"maas-api-revoke-sub-93a027b3-m87mq\" (UID: \"f20bf903-25ba-4be2-8c17-2a10a0255e13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a027b3-m87mq" Nov 02 01:44:36.582444 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.575609 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f20bf903-25ba-4be2-8c17-2a10a0255e13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93a027b3-m87mq\" (UID: \"f20bf903-25ba-4be2-8c17-2a10a0255e13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a027b3-m87mq" Nov 02 01:44:36.676875 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.676838 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f20bf903-25ba-4be2-8c17-2a10a0255e13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93a027b3-m87mq\" (UID: \"f20bf903-25ba-4be2-8c17-2a10a0255e13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a027b3-m87mq" Nov 02 01:44:36.677098 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.676945 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpzfm\" (UniqueName: \"kubernetes.io/projected/f20bf903-25ba-4be2-8c17-2a10a0255e13-kube-api-access-tpzfm\") pod \"maas-api-revoke-sub-93a027b3-m87mq\" (UID: \"f20bf903-25ba-4be2-8c17-2a10a0255e13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a027b3-m87mq" Nov 02 01:44:36.677219 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.677198 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f20bf903-25ba-4be2-8c17-2a10a0255e13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93a027b3-m87mq\" (UID: \"f20bf903-25ba-4be2-8c17-2a10a0255e13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a027b3-m87mq" Nov 02 01:44:36.677635 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.677606 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f20bf903-25ba-4be2-8c17-2a10a0255e13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93a027b3-m87mq\" (UID: \"f20bf903-25ba-4be2-8c17-2a10a0255e13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a027b3-m87mq" Nov 02 01:44:36.680011 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.679980 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f20bf903-25ba-4be2-8c17-2a10a0255e13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93a027b3-m87mq\" (UID: \"f20bf903-25ba-4be2-8c17-2a10a0255e13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a027b3-m87mq" Nov 02 01:44:36.689616 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.689591 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpzfm\" (UniqueName: \"kubernetes.io/projected/f20bf903-25ba-4be2-8c17-2a10a0255e13-kube-api-access-tpzfm\") pod \"maas-api-revoke-sub-93a027b3-m87mq\" (UID: \"f20bf903-25ba-4be2-8c17-2a10a0255e13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a027b3-m87mq" Nov 02 01:44:36.968513 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:36.968432 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a027b3-m87mq" Nov 02 01:44:37.259503 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:44:37.259462 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf20bf903_25ba_4be2_8c17_2a10a0255e13.slice/crio-73b1815493c587e964833f718d4fc582bac3ff2c8761b7cbf4aa6fedc34ed480 WatchSource:0}: Error finding container 73b1815493c587e964833f718d4fc582bac3ff2c8761b7cbf4aa6fedc34ed480: Status 404 returned error can't find the container with id 73b1815493c587e964833f718d4fc582bac3ff2c8761b7cbf4aa6fedc34ed480 Nov 02 01:44:37.260030 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:37.260005 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93a027b3-m87mq"] Nov 02 01:44:38.028893 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:38.028790 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a027b3-m87mq" event={"ID":"f20bf903-25ba-4be2-8c17-2a10a0255e13","Type":"ContainerStarted","Data":"dc23a0c381202a1df8b4d96ca6779deadd44781a45c5207c9f4eb9e8333d1e55"} Nov 02 01:44:38.028893 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:38.028883 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a027b3-m87mq" event={"ID":"f20bf903-25ba-4be2-8c17-2a10a0255e13","Type":"ContainerStarted","Data":"73b1815493c587e964833f718d4fc582bac3ff2c8761b7cbf4aa6fedc34ed480"} Nov 02 01:44:38.050031 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:38.049977 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a027b3-m87mq" podStartSLOduration=2.049964436 podStartE2EDuration="2.049964436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:44:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:44:38.046546707 +0000 UTC m=+1782.667551935" watchObservedRunningTime="2026-11-02 01:44:38.049964436 +0000 UTC m=+1782.670969646" Nov 02 01:44:39.040295 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:39.040262 4427 generic.go:356] "Generic (PLEG): container finished" podID="f20bf903-25ba-4be2-8c17-2a10a0255e13" containerID="dc23a0c381202a1df8b4d96ca6779deadd44781a45c5207c9f4eb9e8333d1e55" exitCode=0 Nov 02 01:44:39.040807 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:39.040317 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a027b3-m87mq" event={"ID":"f20bf903-25ba-4be2-8c17-2a10a0255e13","Type":"ContainerDied","Data":"dc23a0c381202a1df8b4d96ca6779deadd44781a45c5207c9f4eb9e8333d1e55"} Nov 02 01:44:40.509825 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:40.509794 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a027b3-m87mq" Nov 02 01:44:40.570716 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:40.570616 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f20bf903-25ba-4be2-8c17-2a10a0255e13-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f20bf903-25ba-4be2-8c17-2a10a0255e13-maas-api-xks-ca\") pod \"f20bf903-25ba-4be2-8c17-2a10a0255e13\" (UID: \"f20bf903-25ba-4be2-8c17-2a10a0255e13\") " Nov 02 01:44:40.570870 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:40.570846 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f20bf903-25ba-4be2-8c17-2a10a0255e13-kube-api-access-tpzfm\" (UniqueName: \"kubernetes.io/projected/f20bf903-25ba-4be2-8c17-2a10a0255e13-kube-api-access-tpzfm\") pod \"f20bf903-25ba-4be2-8c17-2a10a0255e13\" (UID: \"f20bf903-25ba-4be2-8c17-2a10a0255e13\") " Nov 02 01:44:40.570941 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:40.570913 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f20bf903-25ba-4be2-8c17-2a10a0255e13-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f20bf903-25ba-4be2-8c17-2a10a0255e13-maas-api-openshift-ca\") pod \"f20bf903-25ba-4be2-8c17-2a10a0255e13\" (UID: \"f20bf903-25ba-4be2-8c17-2a10a0255e13\") " Nov 02 01:44:40.571308 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:40.571278 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f20bf903-25ba-4be2-8c17-2a10a0255e13-maas-api-openshift-ca" pod "f20bf903-25ba-4be2-8c17-2a10a0255e13" (UID: "f20bf903-25ba-4be2-8c17-2a10a0255e13"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:44:40.571535 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:40.571515 4427 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f20bf903-25ba-4be2-8c17-2a10a0255e13-maas-api-openshift-ca\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:40.572952 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:40.572922 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f20bf903-25ba-4be2-8c17-2a10a0255e13-maas-api-xks-ca" pod "f20bf903-25ba-4be2-8c17-2a10a0255e13" (UID: "f20bf903-25ba-4be2-8c17-2a10a0255e13"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:44:40.573195 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:40.573173 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f20bf903-25ba-4be2-8c17-2a10a0255e13-kube-api-access-tpzfm" pod "f20bf903-25ba-4be2-8c17-2a10a0255e13" (UID: "f20bf903-25ba-4be2-8c17-2a10a0255e13"). InnerVolumeSpecName "kube-api-access-tpzfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:44:40.673236 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:40.673204 4427 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f20bf903-25ba-4be2-8c17-2a10a0255e13-maas-api-xks-ca\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:40.673236 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:40.673237 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpzfm\" (UniqueName: \"kubernetes.io/projected/f20bf903-25ba-4be2-8c17-2a10a0255e13-kube-api-access-tpzfm\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:40.989926 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:40.989881 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4"] Nov 02 01:44:40.990142 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:40.990096 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4" podUID="9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9" containerName="istio-proxy" containerID="cri-o://3568ce28623faffcd65b62ab78a362f36ed1388d24cb534191d8cc7ce3f7c87b" gracePeriod=30 Nov 02 01:44:41.065293 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:41.065261 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a027b3-m87mq" Nov 02 01:44:41.065479 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:41.065295 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a027b3-m87mq" event={"ID":"f20bf903-25ba-4be2-8c17-2a10a0255e13","Type":"ContainerDied","Data":"73b1815493c587e964833f718d4fc582bac3ff2c8761b7cbf4aa6fedc34ed480"} Nov 02 01:44:41.065479 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:41.065348 4427 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73b1815493c587e964833f718d4fc582bac3ff2c8761b7cbf4aa6fedc34ed480" Nov 02 01:44:43.256836 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:43.256779 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-b6ec07-d4f775449-gjvzd" podUID="7ad049fe-74e6-4083-92ed-09db1fb92ada" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 01:44:44.986159 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:44.986118 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0f32d6-647fd89d8c-v79jr"] Nov 02 01:44:44.986771 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:44.986756 4427 cpu_manager.go:414] "containerMap: removing container" podUID="f20bf903-25ba-4be2-8c17-2a10a0255e13" containerName="revoke-keys" Nov 02 01:44:44.986820 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:44.986773 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f20bf903-25ba-4be2-8c17-2a10a0255e13" containerName="revoke-keys" Nov 02 01:44:44.986820 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:44.986805 4427 cpu_manager.go:414] "containerMap: removing container" podUID="e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0" containerName="istio-proxy" Nov 02 01:44:44.986820 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:44.986810 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0" containerName="istio-proxy" Nov 02 01:44:44.986942 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:44.986912 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="c227e450-ed88-4cb4-a5fd-e88f150d14ab" containerName="istio-proxy" Nov 02 01:44:44.986942 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:44.986924 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="51776e18-2d31-454f-8c33-d7fbb169c601" containerName="istio-proxy" Nov 02 01:44:44.986942 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:44.986932 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="e32db4d5-2c23-4d7a-8a3d-ec7866cb35a0" containerName="istio-proxy" Nov 02 01:44:44.986942 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:44.986942 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="f20bf903-25ba-4be2-8c17-2a10a0255e13" containerName="revoke-keys" Nov 02 01:44:44.990789 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:44.990768 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0f32d6-647fd89d8c-v79jr" Nov 02 01:44:44.999632 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:44.999604 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-0f32d6-dockercfg-wnkqt\"" Nov 02 01:44:44.999771 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:44.999702 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-0f32d6\"" Nov 02 01:44:45.003915 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:45.003886 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0f32d6-647fd89d8c-v79jr"] Nov 02 01:44:45.088306 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:45.088205 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17e05811-e6f0-4c5b-be43-d39429d1af11-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-0f32d6-647fd89d8c-v79jr\" (UID: \"17e05811-e6f0-4c5b-be43-d39429d1af11\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0f32d6-647fd89d8c-v79jr" Nov 02 01:44:45.088783 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:45.088754 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psqvd\" (UniqueName: \"kubernetes.io/projected/17e05811-e6f0-4c5b-be43-d39429d1af11-kube-api-access-psqvd\") pod \"payload-pre-processing-e2e-shared-b-w5-0f32d6-647fd89d8c-v79jr\" (UID: \"17e05811-e6f0-4c5b-be43-d39429d1af11\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0f32d6-647fd89d8c-v79jr" Nov 02 01:44:45.191432 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:45.191331 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psqvd\" (UniqueName: \"kubernetes.io/projected/17e05811-e6f0-4c5b-be43-d39429d1af11-kube-api-access-psqvd\") pod \"payload-pre-processing-e2e-shared-b-w5-0f32d6-647fd89d8c-v79jr\" (UID: \"17e05811-e6f0-4c5b-be43-d39429d1af11\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0f32d6-647fd89d8c-v79jr" Nov 02 01:44:45.191655 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:45.191630 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17e05811-e6f0-4c5b-be43-d39429d1af11-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-0f32d6-647fd89d8c-v79jr\" (UID: \"17e05811-e6f0-4c5b-be43-d39429d1af11\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0f32d6-647fd89d8c-v79jr" Nov 02 01:44:45.192893 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:45.192869 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17e05811-e6f0-4c5b-be43-d39429d1af11-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-0f32d6-647fd89d8c-v79jr\" (UID: \"17e05811-e6f0-4c5b-be43-d39429d1af11\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0f32d6-647fd89d8c-v79jr" Nov 02 01:44:45.206170 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:45.206097 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psqvd\" (UniqueName: \"kubernetes.io/projected/17e05811-e6f0-4c5b-be43-d39429d1af11-kube-api-access-psqvd\") pod \"payload-pre-processing-e2e-shared-b-w5-0f32d6-647fd89d8c-v79jr\" (UID: \"17e05811-e6f0-4c5b-be43-d39429d1af11\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0f32d6-647fd89d8c-v79jr" Nov 02 01:44:45.312835 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:45.312746 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0f32d6-647fd89d8c-v79jr" Nov 02 01:44:45.728945 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:45.728903 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0f32d6-647fd89d8c-v79jr"] Nov 02 01:44:46.281974 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:46.281586 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0f32d6-647fd89d8c-v79jr" event={"ID":"17e05811-e6f0-4c5b-be43-d39429d1af11","Type":"ContainerStarted","Data":"4f20ca6d1ed8b5e95fc91ddac56434196c7a2daae0e290eb30bf9d6e6a38f59c"} Nov 02 01:44:46.281974 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:46.281651 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0f32d6-647fd89d8c-v79jr" event={"ID":"17e05811-e6f0-4c5b-be43-d39429d1af11","Type":"ContainerStarted","Data":"f75840cf0220d6694f134f4e6a1758d000d9b05268887e4bdbf00253c2b78cab"} Nov 02 01:44:46.281974 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:46.281701 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0f32d6-647fd89d8c-v79jr" Nov 02 01:44:46.286459 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:46.284934 4427 generic.go:356] "Generic (PLEG): container finished" podID="9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9" containerID="3568ce28623faffcd65b62ab78a362f36ed1388d24cb534191d8cc7ce3f7c87b" exitCode=0 Nov 02 01:44:46.286459 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:46.285061 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4" event={"ID":"9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9","Type":"ContainerDied","Data":"3568ce28623faffcd65b62ab78a362f36ed1388d24cb534191d8cc7ce3f7c87b"} Nov 02 01:44:46.517315 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:46.517280 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4" Nov 02 01:44:46.548900 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:46.548774 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0f32d6-647fd89d8c-v79jr" podStartSLOduration=2.548754509 podStartE2EDuration="2.548754509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:44:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:44:46.303762956 +0000 UTC m=+1790.924768166" watchObservedRunningTime="2026-11-02 01:44:46.548754509 +0000 UTC m=+1791.169759722" Nov 02 01:44:46.578946 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:46.578903 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9-istio-token\" (UniqueName: \"kubernetes.io/projected/9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9-istio-token\") pod \"9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9\" (UID: \"9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9\") " Nov 02 01:44:46.579132 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:46.579034 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9-kube-api-access-pssnc\" (UniqueName: \"kubernetes.io/projected/9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9-kube-api-access-pssnc\") pod \"9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9\" (UID: \"9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9\") " Nov 02 01:44:46.579132 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:46.579068 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9-istiod-ca-cert\") pod \"9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9\" (UID: \"9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9\") " Nov 02 01:44:46.579132 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:46.579106 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9-workload-socket\") pod \"9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9\" (UID: \"9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9\") " Nov 02 01:44:46.579318 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:46.579132 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9-istio-envoy\") pod \"9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9\" (UID: \"9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9\") " Nov 02 01:44:46.579318 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:46.579164 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9-credential-socket\") pod \"9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9\" (UID: \"9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9\") " Nov 02 01:44:46.579522 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:46.579331 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9-istio-data\") pod \"9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9\" (UID: \"9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9\") " Nov 02 01:44:46.579583 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:46.579559 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9-istio-podinfo\") pod \"9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9\" (UID: \"9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9\") " Nov 02 01:44:46.579638 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:46.579567 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9-istiod-ca-cert" pod "9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9" (UID: "9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:44:46.579638 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:46.579586 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9-workload-certs\") pod \"9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9\" (UID: \"9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9\") " Nov 02 01:44:46.579930 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:46.579901 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9-istio-data" pod "9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9" (UID: "9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:46.580150 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:46.580129 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9-workload-certs" pod "9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9" (UID: "9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:46.580423 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:46.580368 4427 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9-workload-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:46.580423 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:46.580411 4427 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9-istiod-ca-cert\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:46.580568 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:46.580425 4427 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9-istio-data\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:46.580568 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:46.580546 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9-credential-socket" pod "9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9" (UID: "9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:46.582595 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:46.581181 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9-workload-socket" pod "9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9" (UID: "9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:46.582595 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:46.582427 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9-istio-envoy" pod "9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9" (UID: "9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:46.583776 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:46.583747 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9-istio-token" pod "9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9" (UID: "9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:44:46.583936 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:46.583913 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9-kube-api-access-pssnc" pod "9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9" (UID: "9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9"). InnerVolumeSpecName "kube-api-access-pssnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:44:46.585108 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:46.585086 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9-istio-podinfo" pod "9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9" (UID: "9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:44:46.681991 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:46.681955 4427 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9-workload-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:46.681991 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:46.681991 4427 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9-istio-envoy\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:46.682270 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:46.682007 4427 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9-credential-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:46.682270 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:46.682019 4427 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9-istio-podinfo\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:46.682270 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:46.682027 4427 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9-istio-token\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:46.682270 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:46.682039 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pssnc\" (UniqueName: \"kubernetes.io/projected/9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9-kube-api-access-pssnc\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:47.093952 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:47.093904 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-b6ec07-d4f775449-gjvzd" Nov 02 01:44:47.193905 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:47.193805 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ad049fe-74e6-4083-92ed-09db1fb92ada-kube-api-access-56m9d\" (UniqueName: \"kubernetes.io/projected/7ad049fe-74e6-4083-92ed-09db1fb92ada-kube-api-access-56m9d\") pod \"7ad049fe-74e6-4083-92ed-09db1fb92ada\" (UID: \"7ad049fe-74e6-4083-92ed-09db1fb92ada\") " Nov 02 01:44:47.194608 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:47.194576 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ad049fe-74e6-4083-92ed-09db1fb92ada-plugins-config-volume" pod "7ad049fe-74e6-4083-92ed-09db1fb92ada" (UID: "7ad049fe-74e6-4083-92ed-09db1fb92ada"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:44:47.194735 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:47.194110 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ad049fe-74e6-4083-92ed-09db1fb92ada-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ad049fe-74e6-4083-92ed-09db1fb92ada-plugins-config-volume\") pod \"7ad049fe-74e6-4083-92ed-09db1fb92ada\" (UID: \"7ad049fe-74e6-4083-92ed-09db1fb92ada\") " Nov 02 01:44:47.195830 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:47.195805 4427 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ad049fe-74e6-4083-92ed-09db1fb92ada-plugins-config-volume\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:47.197714 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:47.197686 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ad049fe-74e6-4083-92ed-09db1fb92ada-kube-api-access-56m9d" pod "7ad049fe-74e6-4083-92ed-09db1fb92ada" (UID: "7ad049fe-74e6-4083-92ed-09db1fb92ada"). InnerVolumeSpecName "kube-api-access-56m9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:44:47.297207 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:47.297175 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56m9d\" (UniqueName: \"kubernetes.io/projected/7ad049fe-74e6-4083-92ed-09db1fb92ada-kube-api-access-56m9d\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:44:47.297782 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:47.297273 4427 generic.go:356] "Generic (PLEG): container finished" podID="7ad049fe-74e6-4083-92ed-09db1fb92ada" containerID="d3363e7b99cf7a6a00ff13e395b4cf3a8c0c978d0dd1e825bf57d06593382b88" exitCode=0 Nov 02 01:44:47.297782 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:47.297534 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-b6ec07-d4f775449-gjvzd" Nov 02 01:44:47.298641 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:47.298615 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-b6ec07-d4f775449-gjvzd" event={"ID":"7ad049fe-74e6-4083-92ed-09db1fb92ada","Type":"ContainerDied","Data":"d3363e7b99cf7a6a00ff13e395b4cf3a8c0c978d0dd1e825bf57d06593382b88"} Nov 02 01:44:47.298814 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:47.298686 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-b6ec07-d4f775449-gjvzd" event={"ID":"7ad049fe-74e6-4083-92ed-09db1fb92ada","Type":"ContainerDied","Data":"b0c613d56bd797022ab4b7a8cb20dc381dcddf26fcb6e90a351c1d9a0ce39f2e"} Nov 02 01:44:47.298814 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:47.298734 4427 scope.go:122] "RemoveContainer" containerID="d3363e7b99cf7a6a00ff13e395b4cf3a8c0c978d0dd1e825bf57d06593382b88" Nov 02 01:44:47.302114 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:47.301595 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4" Nov 02 01:44:47.306207 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:47.304896 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4" event={"ID":"9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9","Type":"ContainerDied","Data":"dcd8992480701c66624cab73d59ab311737b7094b8bb56a8292e97aabfd7f5f8"} Nov 02 01:44:47.319009 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:47.318980 4427 scope.go:122] "RemoveContainer" containerID="d3363e7b99cf7a6a00ff13e395b4cf3a8c0c978d0dd1e825bf57d06593382b88" Nov 02 01:44:47.319427 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:44:47.319357 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3363e7b99cf7a6a00ff13e395b4cf3a8c0c978d0dd1e825bf57d06593382b88\": container with ID starting with d3363e7b99cf7a6a00ff13e395b4cf3a8c0c978d0dd1e825bf57d06593382b88 not found: ID does not exist" containerID="d3363e7b99cf7a6a00ff13e395b4cf3a8c0c978d0dd1e825bf57d06593382b88" Nov 02 01:44:47.319427 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:47.319407 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3363e7b99cf7a6a00ff13e395b4cf3a8c0c978d0dd1e825bf57d06593382b88"} err="failed to get container status \"d3363e7b99cf7a6a00ff13e395b4cf3a8c0c978d0dd1e825bf57d06593382b88\": rpc error: code = NotFound desc = could not find container \"d3363e7b99cf7a6a00ff13e395b4cf3a8c0c978d0dd1e825bf57d06593382b88\": container with ID starting with d3363e7b99cf7a6a00ff13e395b4cf3a8c0c978d0dd1e825bf57d06593382b88 not found: ID does not exist" Nov 02 01:44:47.319427 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:47.319427 4427 scope.go:122] "RemoveContainer" containerID="3568ce28623faffcd65b62ab78a362f36ed1388d24cb534191d8cc7ce3f7c87b" Nov 02 01:44:47.336089 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:47.336058 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b6ec07-d4f775449-gjvzd"] Nov 02 01:44:47.342167 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:47.342139 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b6ec07-d4f775449-gjvzd"] Nov 02 01:44:47.353263 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:47.353233 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4"] Nov 02 01:44:47.361121 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:47.361091 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-2l6q4"] Nov 02 01:44:47.397689 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:47.397649 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp"] Nov 02 01:44:47.400508 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:47.400479 4427 cpu_manager.go:414] "containerMap: removing container" podUID="7ad049fe-74e6-4083-92ed-09db1fb92ada" containerName="payload-processing" Nov 02 01:44:47.400508 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:47.400504 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ad049fe-74e6-4083-92ed-09db1fb92ada" containerName="payload-processing" Nov 02 01:44:47.400730 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:47.400528 4427 cpu_manager.go:414] "containerMap: removing container" podUID="9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9" containerName="istio-proxy" Nov 02 01:44:47.400730 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:47.400538 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9" containerName="istio-proxy" Nov 02 01:44:47.400730 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:47.400674 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9" containerName="istio-proxy" Nov 02 01:44:47.400730 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:47.400701 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ad049fe-74e6-4083-92ed-09db1fb92ada" containerName="payload-processing" Nov 02 01:44:47.441398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:47.441344 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp"] Nov 02 01:44:47.441614 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:47.441508 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp" Nov 02 01:44:47.448933 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:47.448852 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-c8d4a0-openshift-default-dockercfg-qcqst\"" Nov 02 01:44:47.501672 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:47.501634 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d6df883-72f0-4727-acbc-72bb9bd02f05-credential-socket\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp\" (UID: \"1d6df883-72f0-4727-acbc-72bb9bd02f05\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp" Nov 02 01:44:47.501859 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:47.501711 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d6df883-72f0-4727-acbc-72bb9bd02f05-workload-socket\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp\" (UID: \"1d6df883-72f0-4727-acbc-72bb9bd02f05\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp" Nov 02 01:44:47.501859 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:47.501751 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d6df883-72f0-4727-acbc-72bb9bd02f05-istio-podinfo\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp\" (UID: \"1d6df883-72f0-4727-acbc-72bb9bd02f05\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp" Nov 02 01:44:47.501859 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:47.501784 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d6df883-72f0-4727-acbc-72bb9bd02f05-istio-data\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp\" (UID: \"1d6df883-72f0-4727-acbc-72bb9bd02f05\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp" Nov 02 01:44:47.501859 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:47.501851 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d6df883-72f0-4727-acbc-72bb9bd02f05-workload-certs\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp\" (UID: \"1d6df883-72f0-4727-acbc-72bb9bd02f05\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp" Nov 02 01:44:47.502034 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:47.502014 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcj5t\" (UniqueName: \"kubernetes.io/projected/1d6df883-72f0-4727-acbc-72bb9bd02f05-kube-api-access-xcj5t\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp\" (UID: \"1d6df883-72f0-4727-acbc-72bb9bd02f05\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp" Nov 02 01:44:47.502082 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:47.502045 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d6df883-72f0-4727-acbc-72bb9bd02f05-istio-envoy\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp\" (UID: \"1d6df883-72f0-4727-acbc-72bb9bd02f05\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp" Nov 02 01:44:47.502082 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:47.502063 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d6df883-72f0-4727-acbc-72bb9bd02f05-istiod-ca-cert\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp\" (UID: \"1d6df883-72f0-4727-acbc-72bb9bd02f05\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp" Nov 02 01:44:47.502163 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:47.502148 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d6df883-72f0-4727-acbc-72bb9bd02f05-istio-token\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp\" (UID: \"1d6df883-72f0-4727-acbc-72bb9bd02f05\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp" Nov 02 01:44:47.537540 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:47.537505 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ad049fe-74e6-4083-92ed-09db1fb92ada" path="/var/lib/kubelet/pods/7ad049fe-74e6-4083-92ed-09db1fb92ada/volumes" Nov 02 01:44:47.538081 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:47.538062 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9" path="/var/lib/kubelet/pods/9b13ff7b-15dc-4a12-882d-6c0fc8fe25d9/volumes" Nov 02 01:44:47.607820 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:47.606965 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcj5t\" (UniqueName: \"kubernetes.io/projected/1d6df883-72f0-4727-acbc-72bb9bd02f05-kube-api-access-xcj5t\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp\" (UID: \"1d6df883-72f0-4727-acbc-72bb9bd02f05\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp" Nov 02 01:44:47.607820 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:47.607042 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d6df883-72f0-4727-acbc-72bb9bd02f05-istio-envoy\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp\" (UID: \"1d6df883-72f0-4727-acbc-72bb9bd02f05\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp" Nov 02 01:44:47.607820 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:47.607083 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d6df883-72f0-4727-acbc-72bb9bd02f05-istiod-ca-cert\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp\" (UID: \"1d6df883-72f0-4727-acbc-72bb9bd02f05\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp" Nov 02 01:44:47.607820 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:47.607141 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d6df883-72f0-4727-acbc-72bb9bd02f05-istio-token\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp\" (UID: \"1d6df883-72f0-4727-acbc-72bb9bd02f05\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp" Nov 02 01:44:47.619398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:47.616068 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d6df883-72f0-4727-acbc-72bb9bd02f05-istiod-ca-cert\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp\" (UID: \"1d6df883-72f0-4727-acbc-72bb9bd02f05\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp" Nov 02 01:44:47.619398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:47.616569 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d6df883-72f0-4727-acbc-72bb9bd02f05-credential-socket\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp\" (UID: \"1d6df883-72f0-4727-acbc-72bb9bd02f05\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp" Nov 02 01:44:47.619398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:47.616649 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d6df883-72f0-4727-acbc-72bb9bd02f05-workload-socket\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp\" (UID: \"1d6df883-72f0-4727-acbc-72bb9bd02f05\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp" Nov 02 01:44:47.619398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:47.616707 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d6df883-72f0-4727-acbc-72bb9bd02f05-istio-podinfo\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp\" (UID: \"1d6df883-72f0-4727-acbc-72bb9bd02f05\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp" Nov 02 01:44:47.619398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:47.616762 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d6df883-72f0-4727-acbc-72bb9bd02f05-istio-data\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp\" (UID: \"1d6df883-72f0-4727-acbc-72bb9bd02f05\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp" Nov 02 01:44:47.619398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:47.616791 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d6df883-72f0-4727-acbc-72bb9bd02f05-workload-certs\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp\" (UID: \"1d6df883-72f0-4727-acbc-72bb9bd02f05\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp" Nov 02 01:44:47.619398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:47.617261 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d6df883-72f0-4727-acbc-72bb9bd02f05-workload-certs\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp\" (UID: \"1d6df883-72f0-4727-acbc-72bb9bd02f05\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp" Nov 02 01:44:47.619398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:47.617335 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d6df883-72f0-4727-acbc-72bb9bd02f05-workload-socket\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp\" (UID: \"1d6df883-72f0-4727-acbc-72bb9bd02f05\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp" Nov 02 01:44:47.619398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:47.619057 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d6df883-72f0-4727-acbc-72bb9bd02f05-istio-envoy\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp\" (UID: \"1d6df883-72f0-4727-acbc-72bb9bd02f05\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp" Nov 02 01:44:47.619398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:47.619354 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d6df883-72f0-4727-acbc-72bb9bd02f05-istio-data\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp\" (UID: \"1d6df883-72f0-4727-acbc-72bb9bd02f05\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp" Nov 02 01:44:47.627774 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:47.627740 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d6df883-72f0-4727-acbc-72bb9bd02f05-credential-socket\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp\" (UID: \"1d6df883-72f0-4727-acbc-72bb9bd02f05\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp" Nov 02 01:44:47.647362 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:47.647328 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d6df883-72f0-4727-acbc-72bb9bd02f05-istio-token\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp\" (UID: \"1d6df883-72f0-4727-acbc-72bb9bd02f05\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp" Nov 02 01:44:47.648234 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:47.648170 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcj5t\" (UniqueName: \"kubernetes.io/projected/1d6df883-72f0-4727-acbc-72bb9bd02f05-kube-api-access-xcj5t\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp\" (UID: \"1d6df883-72f0-4727-acbc-72bb9bd02f05\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp" Nov 02 01:44:47.648866 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:47.648843 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d6df883-72f0-4727-acbc-72bb9bd02f05-istio-podinfo\") pod \"e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp\" (UID: \"1d6df883-72f0-4727-acbc-72bb9bd02f05\") " pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp" Nov 02 01:44:47.769448 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:47.768658 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp" Nov 02 01:44:48.187323 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:48.187290 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp"] Nov 02 01:44:48.188226 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:44:48.188198 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d6df883_72f0_4727_acbc_72bb9bd02f05.slice/crio-e8a458a7a498a9070ba2c9f8400cddcbbc1b9dc616bf68b3eff64f2f3337c7cf WatchSource:0}: Error finding container e8a458a7a498a9070ba2c9f8400cddcbbc1b9dc616bf68b3eff64f2f3337c7cf: Status 404 returned error can't find the container with id e8a458a7a498a9070ba2c9f8400cddcbbc1b9dc616bf68b3eff64f2f3337c7cf Nov 02 01:44:48.191294 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:48.191263 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:44:48.191360 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:48.191329 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:44:48.191426 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:48.191359 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:44:48.314357 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:48.314313 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp" event={"ID":"1d6df883-72f0-4727-acbc-72bb9bd02f05","Type":"ContainerStarted","Data":"1998796f0577d5eb3cec5d2bcc5e6bcc70971c4401f42cf0951a7e59b1e3cc7c"} Nov 02 01:44:48.314867 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:48.314400 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp" event={"ID":"1d6df883-72f0-4727-acbc-72bb9bd02f05","Type":"ContainerStarted","Data":"e8a458a7a498a9070ba2c9f8400cddcbbc1b9dc616bf68b3eff64f2f3337c7cf"} Nov 02 01:44:48.340740 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:48.340673 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp" podStartSLOduration=1.340652082 podStartE2EDuration="1.340652082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:44:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:44:48.336646931 +0000 UTC m=+1792.957652140" watchObservedRunningTime="2026-11-02 01:44:48.340652082 +0000 UTC m=+1792.961657297" Nov 02 01:44:48.770046 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:48.770004 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp" Nov 02 01:44:49.775919 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:49.775889 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp" Nov 02 01:44:50.332733 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:50.332694 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp" Nov 02 01:44:50.333867 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:50.333841 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp" Nov 02 01:44:52.213504 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:52.213466 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d4a0-6595f44dff-m47jt"] Nov 02 01:44:52.217831 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:52.217807 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d4a0-6595f44dff-m47jt" Nov 02 01:44:52.221877 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:52.221845 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-c8d4a0\"" Nov 02 01:44:52.223111 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:52.223076 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-c8d4a0-dockercfg-swkz7\"" Nov 02 01:44:52.228365 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:52.228343 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d4a0-6595f44dff-m47jt"] Nov 02 01:44:52.246093 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:52.246060 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp6dn\" (UniqueName: \"kubernetes.io/projected/fca29702-c34c-4933-a93a-143c5481ba3a-kube-api-access-sp6dn\") pod \"payload-pre-processing-e2e-shared-b-w0-c8d4a0-6595f44dff-m47jt\" (UID: \"fca29702-c34c-4933-a93a-143c5481ba3a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d4a0-6595f44dff-m47jt" Nov 02 01:44:52.246263 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:52.246167 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fca29702-c34c-4933-a93a-143c5481ba3a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-c8d4a0-6595f44dff-m47jt\" (UID: \"fca29702-c34c-4933-a93a-143c5481ba3a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d4a0-6595f44dff-m47jt" Nov 02 01:44:52.351399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:52.347680 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp6dn\" (UniqueName: \"kubernetes.io/projected/fca29702-c34c-4933-a93a-143c5481ba3a-kube-api-access-sp6dn\") pod \"payload-pre-processing-e2e-shared-b-w0-c8d4a0-6595f44dff-m47jt\" (UID: \"fca29702-c34c-4933-a93a-143c5481ba3a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d4a0-6595f44dff-m47jt" Nov 02 01:44:52.351399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:52.347777 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fca29702-c34c-4933-a93a-143c5481ba3a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-c8d4a0-6595f44dff-m47jt\" (UID: \"fca29702-c34c-4933-a93a-143c5481ba3a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d4a0-6595f44dff-m47jt" Nov 02 01:44:52.351399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:52.348861 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fca29702-c34c-4933-a93a-143c5481ba3a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-c8d4a0-6595f44dff-m47jt\" (UID: \"fca29702-c34c-4933-a93a-143c5481ba3a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d4a0-6595f44dff-m47jt" Nov 02 01:44:52.414403 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:52.413483 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp6dn\" (UniqueName: \"kubernetes.io/projected/fca29702-c34c-4933-a93a-143c5481ba3a-kube-api-access-sp6dn\") pod \"payload-pre-processing-e2e-shared-b-w0-c8d4a0-6595f44dff-m47jt\" (UID: \"fca29702-c34c-4933-a93a-143c5481ba3a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d4a0-6595f44dff-m47jt" Nov 02 01:44:52.535443 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:52.535107 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d4a0-6595f44dff-m47jt" Nov 02 01:44:52.840967 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:52.840932 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d4a0-6595f44dff-m47jt"] Nov 02 01:44:52.841783 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:44:52.841759 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfca29702_c34c_4933_a93a_143c5481ba3a.slice/crio-3fa4ef2c2f34bad9fdbb972fff7de519e18ee24dae029e0e980c5538b84311bd WatchSource:0}: Error finding container 3fa4ef2c2f34bad9fdbb972fff7de519e18ee24dae029e0e980c5538b84311bd: Status 404 returned error can't find the container with id 3fa4ef2c2f34bad9fdbb972fff7de519e18ee24dae029e0e980c5538b84311bd Nov 02 01:44:53.464401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:53.458976 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d4a0-6595f44dff-m47jt" event={"ID":"fca29702-c34c-4933-a93a-143c5481ba3a","Type":"ContainerStarted","Data":"2e3970879f8b3a231c8a1dbb7cc862a378562a3558e2b134903caf126aac381b"} Nov 02 01:44:53.464401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:53.459058 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d4a0-6595f44dff-m47jt" event={"ID":"fca29702-c34c-4933-a93a-143c5481ba3a","Type":"ContainerStarted","Data":"3fa4ef2c2f34bad9fdbb972fff7de519e18ee24dae029e0e980c5538b84311bd"} Nov 02 01:44:53.464401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:53.460295 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d4a0-6595f44dff-m47jt" Nov 02 01:44:53.579098 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:53.577965 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d4a0-6595f44dff-m47jt" podStartSLOduration=1.577945885 podStartE2EDuration="1.577945885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:44:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:44:53.574746908 +0000 UTC m=+1798.195752120" watchObservedRunningTime="2026-11-02 01:44:53.577945885 +0000 UTC m=+1798.198951097" Nov 02 01:44:57.305573 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:57.305541 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0f32d6-647fd89d8c-v79jr" Nov 02 01:44:57.814070 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:57.814035 4427 scope.go:122] "RemoveContainer" containerID="4500bdb77b7beb118e1cecdef9ee744975e5d25d66798934008922007f4ec916" Nov 02 01:44:57.825532 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:57.825501 4427 scope.go:122] "RemoveContainer" containerID="d57cc647a5acf0d4dc1fdbfaf39d66e0cfc2d82b3a7982df3296a3cec576e614" Nov 02 01:44:59.378012 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:59.377971 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8"] Nov 02 01:44:59.378555 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:59.378093 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7"] Nov 02 01:44:59.378555 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:59.378308 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7" podUID="f3730ee2-6ba0-4dec-b103-214881818132" containerName="istio-proxy" containerID="cri-o://a828e3e38efe8596a7047f188634226964ed3fd28db5919d0ae3846282c6744c" gracePeriod=30 Nov 02 01:44:59.378689 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:44:59.378564 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8" podUID="af735a61-695b-4a66-a7cb-d0df06b3b58b" containerName="istio-proxy" containerID="cri-o://40e8a42407d0d5421dd8d43594a03c7f6cdcf21528adc613f5c28d4bf54a2c65" gracePeriod=30 Nov 02 01:45:00.135855 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.135824 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-123986-29893065-dvhsz"] Nov 02 01:45:00.142666 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.142638 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-123986-29893065-dvhsz" Nov 02 01:45:00.157673 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.152121 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\"" Nov 02 01:45:00.158246 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.158216 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bbb385-29893065-wl5kl"] Nov 02 01:45:00.163900 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.163227 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bbb385-29893065-wl5kl" Nov 02 01:45:00.163900 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.163894 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893065-ltmvv"] Nov 02 01:45:00.231287 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.231246 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-123986-29893065-dvhsz"] Nov 02 01:45:00.231546 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.231306 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bbb385-29893065-wl5kl"] Nov 02 01:45:00.231546 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.231336 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893065-ltmvv"] Nov 02 01:45:00.231546 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.231467 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893065-ltmvv" Nov 02 01:45:00.244632 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.244583 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3c4ea5-29893065-d2hkv"] Nov 02 01:45:00.268954 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.268916 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e06f2-29893065-4ql7w"] Nov 02 01:45:00.269811 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.269254 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3c4ea5-29893065-d2hkv" Nov 02 01:45:00.287167 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.282211 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e06f2-29893065-4ql7w" Nov 02 01:45:00.287167 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.283152 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e06f2-29893065-4ql7w"] Nov 02 01:45:00.287167 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.284293 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3c4ea5-29893065-d2hkv"] Nov 02 01:45:00.297570 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.297535 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9d816-29893065-vvz6j"] Nov 02 01:45:00.300484 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.300445 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w48j9\" (UniqueName: \"kubernetes.io/projected/747eccfe-eecb-4cec-99a9-aea7857e1a34-kube-api-access-w48j9\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-bbb385-29893065-wl5kl\" (UID: \"747eccfe-eecb-4cec-99a9-aea7857e1a34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bbb385-29893065-wl5kl" Nov 02 01:45:00.300628 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.300554 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7ebe1a8-c572-4b18-98d9-083b497aa758-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-123986-29893065-dvhsz\" (UID: \"c7ebe1a8-c572-4b18-98d9-083b497aa758\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-123986-29893065-dvhsz" Nov 02 01:45:00.300845 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.300826 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/747eccfe-eecb-4cec-99a9-aea7857e1a34-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-bbb385-29893065-wl5kl\" (UID: \"747eccfe-eecb-4cec-99a9-aea7857e1a34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bbb385-29893065-wl5kl" Nov 02 01:45:00.300959 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.300943 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgqk7\" (UniqueName: \"kubernetes.io/projected/c7ebe1a8-c572-4b18-98d9-083b497aa758-kube-api-access-wgqk7\") pod \"maas-api-key-cleanup-e2e-worker-w5-123986-29893065-dvhsz\" (UID: \"c7ebe1a8-c572-4b18-98d9-083b497aa758\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-123986-29893065-dvhsz" Nov 02 01:45:00.315209 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.315070 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-00f1da-29893065-vkvdj"] Nov 02 01:45:00.318253 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.318228 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9d816-29893065-vvz6j" Nov 02 01:45:00.320489 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.320467 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0f32d6-29893065-5kg4t"] Nov 02 01:45:00.320647 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.320627 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-00f1da-29893065-vkvdj" Nov 02 01:45:00.342631 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.342597 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0f32d6-29893065-5kg4t"] Nov 02 01:45:00.342802 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.342656 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9d816-29893065-vvz6j"] Nov 02 01:45:00.342802 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.342770 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0f32d6-29893065-5kg4t" Nov 02 01:45:00.347664 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.347632 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-00f1da-29893065-vkvdj"] Nov 02 01:45:00.402145 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.402049 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkt5q\" (UniqueName: \"kubernetes.io/projected/d983a998-461c-4341-aea9-1eb65375bbb7-kube-api-access-hkt5q\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-0e06f2-29893065-4ql7w\" (UID: \"d983a998-461c-4341-aea9-1eb65375bbb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e06f2-29893065-4ql7w" Nov 02 01:45:00.402656 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.402158 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/747eccfe-eecb-4cec-99a9-aea7857e1a34-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-bbb385-29893065-wl5kl\" (UID: \"747eccfe-eecb-4cec-99a9-aea7857e1a34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bbb385-29893065-wl5kl" Nov 02 01:45:00.402656 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.402207 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvm8x\" (UniqueName: \"kubernetes.io/projected/bcd48699-8090-4574-934c-b29668788243-kube-api-access-cvm8x\") pod \"maas-api-key-cleanup-e2e-worker-w6-b9d816-29893065-vvz6j\" (UID: \"bcd48699-8090-4574-934c-b29668788243\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9d816-29893065-vvz6j" Nov 02 01:45:00.402656 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.402303 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgqk7\" (UniqueName: \"kubernetes.io/projected/c7ebe1a8-c572-4b18-98d9-083b497aa758-kube-api-access-wgqk7\") pod \"maas-api-key-cleanup-e2e-worker-w5-123986-29893065-dvhsz\" (UID: \"c7ebe1a8-c572-4b18-98d9-083b497aa758\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-123986-29893065-dvhsz" Nov 02 01:45:00.402656 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.402420 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/635a1467-2a08-4b8b-a8c0-b16e2a361888-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893065-ltmvv\" (UID: \"635a1467-2a08-4b8b-a8c0-b16e2a361888\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893065-ltmvv" Nov 02 01:45:00.402656 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.402469 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w48j9\" (UniqueName: \"kubernetes.io/projected/747eccfe-eecb-4cec-99a9-aea7857e1a34-kube-api-access-w48j9\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-bbb385-29893065-wl5kl\" (UID: \"747eccfe-eecb-4cec-99a9-aea7857e1a34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bbb385-29893065-wl5kl" Nov 02 01:45:00.402656 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.402558 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d983a998-461c-4341-aea9-1eb65375bbb7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-0e06f2-29893065-4ql7w\" (UID: \"d983a998-461c-4341-aea9-1eb65375bbb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e06f2-29893065-4ql7w" Nov 02 01:45:00.402656 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.402605 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7ebe1a8-c572-4b18-98d9-083b497aa758-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-123986-29893065-dvhsz\" (UID: \"c7ebe1a8-c572-4b18-98d9-083b497aa758\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-123986-29893065-dvhsz" Nov 02 01:45:00.402656 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.402638 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4xhd\" (UniqueName: \"kubernetes.io/projected/886fa3e6-9143-4823-903d-7e68d06100ee-kube-api-access-j4xhd\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-3c4ea5-29893065-d2hkv\" (UID: \"886fa3e6-9143-4823-903d-7e68d06100ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3c4ea5-29893065-d2hkv" Nov 02 01:45:00.403099 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.402712 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5779a226-eaa5-4428-99d1-47e4936591a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-00f1da-29893065-vkvdj\" (UID: \"5779a226-eaa5-4428-99d1-47e4936591a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-00f1da-29893065-vkvdj" Nov 02 01:45:00.403099 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.402848 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcd48699-8090-4574-934c-b29668788243-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-b9d816-29893065-vvz6j\" (UID: \"bcd48699-8090-4574-934c-b29668788243\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9d816-29893065-vvz6j" Nov 02 01:45:00.403099 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.402910 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/886fa3e6-9143-4823-903d-7e68d06100ee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-3c4ea5-29893065-d2hkv\" (UID: \"886fa3e6-9143-4823-903d-7e68d06100ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3c4ea5-29893065-d2hkv" Nov 02 01:45:00.403099 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.402986 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/747eccfe-eecb-4cec-99a9-aea7857e1a34-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-bbb385-29893065-wl5kl\" (UID: \"747eccfe-eecb-4cec-99a9-aea7857e1a34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bbb385-29893065-wl5kl" Nov 02 01:45:00.403099 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.403012 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5gdd\" (UniqueName: \"kubernetes.io/projected/5779a226-eaa5-4428-99d1-47e4936591a2-kube-api-access-p5gdd\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-00f1da-29893065-vkvdj\" (UID: \"5779a226-eaa5-4428-99d1-47e4936591a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-00f1da-29893065-vkvdj" Nov 02 01:45:00.403099 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.403090 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm9nd\" (UniqueName: \"kubernetes.io/projected/635a1467-2a08-4b8b-a8c0-b16e2a361888-kube-api-access-rm9nd\") pod \"maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893065-ltmvv\" (UID: \"635a1467-2a08-4b8b-a8c0-b16e2a361888\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893065-ltmvv" Nov 02 01:45:00.403440 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.403263 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7ebe1a8-c572-4b18-98d9-083b497aa758-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-123986-29893065-dvhsz\" (UID: \"c7ebe1a8-c572-4b18-98d9-083b497aa758\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-123986-29893065-dvhsz" Nov 02 01:45:00.426448 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.426415 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgqk7\" (UniqueName: \"kubernetes.io/projected/c7ebe1a8-c572-4b18-98d9-083b497aa758-kube-api-access-wgqk7\") pod \"maas-api-key-cleanup-e2e-worker-w5-123986-29893065-dvhsz\" (UID: \"c7ebe1a8-c572-4b18-98d9-083b497aa758\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-123986-29893065-dvhsz" Nov 02 01:45:00.427337 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.427305 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w48j9\" (UniqueName: \"kubernetes.io/projected/747eccfe-eecb-4cec-99a9-aea7857e1a34-kube-api-access-w48j9\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-bbb385-29893065-wl5kl\" (UID: \"747eccfe-eecb-4cec-99a9-aea7857e1a34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bbb385-29893065-wl5kl" Nov 02 01:45:00.460585 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.460554 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-123986-29893065-dvhsz" Nov 02 01:45:00.504894 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.504860 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvm8x\" (UniqueName: \"kubernetes.io/projected/bcd48699-8090-4574-934c-b29668788243-kube-api-access-cvm8x\") pod \"maas-api-key-cleanup-e2e-worker-w6-b9d816-29893065-vvz6j\" (UID: \"bcd48699-8090-4574-934c-b29668788243\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9d816-29893065-vvz6j" Nov 02 01:45:00.505086 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.505031 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/635a1467-2a08-4b8b-a8c0-b16e2a361888-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893065-ltmvv\" (UID: \"635a1467-2a08-4b8b-a8c0-b16e2a361888\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893065-ltmvv" Nov 02 01:45:00.505159 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.505087 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j26x6\" (UniqueName: \"kubernetes.io/projected/10bfbdc2-73c3-43b5-9087-bc07be95319e-kube-api-access-j26x6\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-0f32d6-29893065-5kg4t\" (UID: \"10bfbdc2-73c3-43b5-9087-bc07be95319e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0f32d6-29893065-5kg4t" Nov 02 01:45:00.505159 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.505118 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d983a998-461c-4341-aea9-1eb65375bbb7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-0e06f2-29893065-4ql7w\" (UID: \"d983a998-461c-4341-aea9-1eb65375bbb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e06f2-29893065-4ql7w" Nov 02 01:45:00.505264 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.505159 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4xhd\" (UniqueName: \"kubernetes.io/projected/886fa3e6-9143-4823-903d-7e68d06100ee-kube-api-access-j4xhd\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-3c4ea5-29893065-d2hkv\" (UID: \"886fa3e6-9143-4823-903d-7e68d06100ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3c4ea5-29893065-d2hkv" Nov 02 01:45:00.505495 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.505462 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5779a226-eaa5-4428-99d1-47e4936591a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-00f1da-29893065-vkvdj\" (UID: \"5779a226-eaa5-4428-99d1-47e4936591a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-00f1da-29893065-vkvdj" Nov 02 01:45:00.505691 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.505555 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcd48699-8090-4574-934c-b29668788243-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-b9d816-29893065-vvz6j\" (UID: \"bcd48699-8090-4574-934c-b29668788243\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9d816-29893065-vvz6j" Nov 02 01:45:00.505691 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.505608 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10bfbdc2-73c3-43b5-9087-bc07be95319e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-0f32d6-29893065-5kg4t\" (UID: \"10bfbdc2-73c3-43b5-9087-bc07be95319e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0f32d6-29893065-5kg4t" Nov 02 01:45:00.505691 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.505629 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/886fa3e6-9143-4823-903d-7e68d06100ee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-3c4ea5-29893065-d2hkv\" (UID: \"886fa3e6-9143-4823-903d-7e68d06100ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3c4ea5-29893065-d2hkv" Nov 02 01:45:00.505691 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.505669 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5gdd\" (UniqueName: \"kubernetes.io/projected/5779a226-eaa5-4428-99d1-47e4936591a2-kube-api-access-p5gdd\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-00f1da-29893065-vkvdj\" (UID: \"5779a226-eaa5-4428-99d1-47e4936591a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-00f1da-29893065-vkvdj" Nov 02 01:45:00.505914 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.505732 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm9nd\" (UniqueName: \"kubernetes.io/projected/635a1467-2a08-4b8b-a8c0-b16e2a361888-kube-api-access-rm9nd\") pod \"maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893065-ltmvv\" (UID: \"635a1467-2a08-4b8b-a8c0-b16e2a361888\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893065-ltmvv" Nov 02 01:45:00.505914 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.505780 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkt5q\" (UniqueName: \"kubernetes.io/projected/d983a998-461c-4341-aea9-1eb65375bbb7-kube-api-access-hkt5q\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-0e06f2-29893065-4ql7w\" (UID: \"d983a998-461c-4341-aea9-1eb65375bbb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e06f2-29893065-4ql7w" Nov 02 01:45:00.506050 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.505941 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/635a1467-2a08-4b8b-a8c0-b16e2a361888-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893065-ltmvv\" (UID: \"635a1467-2a08-4b8b-a8c0-b16e2a361888\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893065-ltmvv" Nov 02 01:45:00.506662 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.506636 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d983a998-461c-4341-aea9-1eb65375bbb7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-0e06f2-29893065-4ql7w\" (UID: \"d983a998-461c-4341-aea9-1eb65375bbb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e06f2-29893065-4ql7w" Nov 02 01:45:00.506796 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.506694 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcd48699-8090-4574-934c-b29668788243-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-b9d816-29893065-vvz6j\" (UID: \"bcd48699-8090-4574-934c-b29668788243\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9d816-29893065-vvz6j" Nov 02 01:45:00.507356 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.507332 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5779a226-eaa5-4428-99d1-47e4936591a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-00f1da-29893065-vkvdj\" (UID: \"5779a226-eaa5-4428-99d1-47e4936591a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-00f1da-29893065-vkvdj" Nov 02 01:45:00.507492 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.507418 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/886fa3e6-9143-4823-903d-7e68d06100ee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-3c4ea5-29893065-d2hkv\" (UID: \"886fa3e6-9143-4823-903d-7e68d06100ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3c4ea5-29893065-d2hkv" Nov 02 01:45:00.512552 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.512524 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bbb385-29893065-wl5kl" Nov 02 01:45:00.518347 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.518319 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvm8x\" (UniqueName: \"kubernetes.io/projected/bcd48699-8090-4574-934c-b29668788243-kube-api-access-cvm8x\") pod \"maas-api-key-cleanup-e2e-worker-w6-b9d816-29893065-vvz6j\" (UID: \"bcd48699-8090-4574-934c-b29668788243\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9d816-29893065-vvz6j" Nov 02 01:45:00.519163 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.519113 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5gdd\" (UniqueName: \"kubernetes.io/projected/5779a226-eaa5-4428-99d1-47e4936591a2-kube-api-access-p5gdd\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-00f1da-29893065-vkvdj\" (UID: \"5779a226-eaa5-4428-99d1-47e4936591a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-00f1da-29893065-vkvdj" Nov 02 01:45:00.519329 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.519298 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm9nd\" (UniqueName: \"kubernetes.io/projected/635a1467-2a08-4b8b-a8c0-b16e2a361888-kube-api-access-rm9nd\") pod \"maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893065-ltmvv\" (UID: \"635a1467-2a08-4b8b-a8c0-b16e2a361888\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893065-ltmvv" Nov 02 01:45:00.520764 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.520743 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkt5q\" (UniqueName: \"kubernetes.io/projected/d983a998-461c-4341-aea9-1eb65375bbb7-kube-api-access-hkt5q\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-0e06f2-29893065-4ql7w\" (UID: \"d983a998-461c-4341-aea9-1eb65375bbb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e06f2-29893065-4ql7w" Nov 02 01:45:00.527977 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.526204 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4xhd\" (UniqueName: \"kubernetes.io/projected/886fa3e6-9143-4823-903d-7e68d06100ee-kube-api-access-j4xhd\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-3c4ea5-29893065-d2hkv\" (UID: \"886fa3e6-9143-4823-903d-7e68d06100ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3c4ea5-29893065-d2hkv" Nov 02 01:45:00.611292 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.611253 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j26x6\" (UniqueName: \"kubernetes.io/projected/10bfbdc2-73c3-43b5-9087-bc07be95319e-kube-api-access-j26x6\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-0f32d6-29893065-5kg4t\" (UID: \"10bfbdc2-73c3-43b5-9087-bc07be95319e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0f32d6-29893065-5kg4t" Nov 02 01:45:00.611555 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.611527 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10bfbdc2-73c3-43b5-9087-bc07be95319e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-0f32d6-29893065-5kg4t\" (UID: \"10bfbdc2-73c3-43b5-9087-bc07be95319e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0f32d6-29893065-5kg4t" Nov 02 01:45:00.613026 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.612997 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10bfbdc2-73c3-43b5-9087-bc07be95319e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-0f32d6-29893065-5kg4t\" (UID: \"10bfbdc2-73c3-43b5-9087-bc07be95319e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0f32d6-29893065-5kg4t" Nov 02 01:45:00.613445 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.613421 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893065-ltmvv" Nov 02 01:45:00.626063 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.626036 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j26x6\" (UniqueName: \"kubernetes.io/projected/10bfbdc2-73c3-43b5-9087-bc07be95319e-kube-api-access-j26x6\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-0f32d6-29893065-5kg4t\" (UID: \"10bfbdc2-73c3-43b5-9087-bc07be95319e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0f32d6-29893065-5kg4t" Nov 02 01:45:00.630128 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.629923 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3c4ea5-29893065-d2hkv" Nov 02 01:45:00.654035 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.653768 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e06f2-29893065-4ql7w" Nov 02 01:45:00.662330 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.662302 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9d816-29893065-vvz6j" Nov 02 01:45:00.672875 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.672679 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-00f1da-29893065-vkvdj" Nov 02 01:45:00.681444 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.680534 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0f32d6-29893065-5kg4t" Nov 02 01:45:00.789942 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.779213 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-123986-29893065-dvhsz"] Nov 02 01:45:00.909042 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:45:00.907329 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod747eccfe_eecb_4cec_99a9_aea7857e1a34.slice/crio-af6b5a0758b80dffc44f7cc4f33c4d925cd28c49a44de509a37351d8fc0347b6 WatchSource:0}: Error finding container af6b5a0758b80dffc44f7cc4f33c4d925cd28c49a44de509a37351d8fc0347b6: Status 404 returned error can't find the container with id af6b5a0758b80dffc44f7cc4f33c4d925cd28c49a44de509a37351d8fc0347b6 Nov 02 01:45:00.935698 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.935670 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bbb385-29893065-wl5kl"] Nov 02 01:45:00.959406 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.958279 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893065-ltmvv"] Nov 02 01:45:00.973745 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:45:00.973238 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod635a1467_2a08_4b8b_a8c0_b16e2a361888.slice/crio-e5d68a3b98a0eddeba23af77cfbdff705206f25d56573471377c1a933e2c27c1 WatchSource:0}: Error finding container e5d68a3b98a0eddeba23af77cfbdff705206f25d56573471377c1a933e2c27c1: Status 404 returned error can't find the container with id e5d68a3b98a0eddeba23af77cfbdff705206f25d56573471377c1a933e2c27c1 Nov 02 01:45:00.988024 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:00.987997 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3c4ea5-29893065-d2hkv"] Nov 02 01:45:01.063751 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:45:01.063716 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod886fa3e6_9143_4823_903d_7e68d06100ee.slice/crio-7f5528f3813d7e603ddabf9af36f6de066308994f86dfbc3e28d5a042eb442a2 WatchSource:0}: Error finding container 7f5528f3813d7e603ddabf9af36f6de066308994f86dfbc3e28d5a042eb442a2: Status 404 returned error can't find the container with id 7f5528f3813d7e603ddabf9af36f6de066308994f86dfbc3e28d5a042eb442a2 Nov 02 01:45:01.207073 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:01.207039 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-00f1da-29893065-vkvdj"] Nov 02 01:45:01.226702 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:01.226675 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e06f2-29893065-4ql7w"] Nov 02 01:45:01.257062 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:01.257036 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9d816-29893065-vvz6j"] Nov 02 01:45:01.260942 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:01.260917 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0f32d6-29893065-5kg4t"] Nov 02 01:45:01.265773 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:45:01.265748 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5779a226_eaa5_4428_99d1_47e4936591a2.slice/crio-9cffdc73841c9ac4b4022998f38c8108cf16e54b0fc9ab892c9142971d51098f WatchSource:0}: Error finding container 9cffdc73841c9ac4b4022998f38c8108cf16e54b0fc9ab892c9142971d51098f: Status 404 returned error can't find the container with id 9cffdc73841c9ac4b4022998f38c8108cf16e54b0fc9ab892c9142971d51098f Nov 02 01:45:01.266769 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:45:01.266634 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd983a998_461c_4341_aea9_1eb65375bbb7.slice/crio-59df0cffd315d944f8acf17d9881406903fa197cb5b8a9292213108ddc14d6b3 WatchSource:0}: Error finding container 59df0cffd315d944f8acf17d9881406903fa197cb5b8a9292213108ddc14d6b3: Status 404 returned error can't find the container with id 59df0cffd315d944f8acf17d9881406903fa197cb5b8a9292213108ddc14d6b3 Nov 02 01:45:01.538127 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:01.538096 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bbb385-29893065-wl5kl" event={"ID":"747eccfe-eecb-4cec-99a9-aea7857e1a34","Type":"ContainerStarted","Data":"af6b5a0758b80dffc44f7cc4f33c4d925cd28c49a44de509a37351d8fc0347b6"} Nov 02 01:45:01.538503 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:01.538146 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0f32d6-29893065-5kg4t" event={"ID":"10bfbdc2-73c3-43b5-9087-bc07be95319e","Type":"ContainerStarted","Data":"9b49c1fa05c0f968c077b1147ad2588061b21a323390a56201c6078ef7a9670f"} Nov 02 01:45:01.538861 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:01.538837 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e06f2-29893065-4ql7w" event={"ID":"d983a998-461c-4341-aea9-1eb65375bbb7","Type":"ContainerStarted","Data":"59df0cffd315d944f8acf17d9881406903fa197cb5b8a9292213108ddc14d6b3"} Nov 02 01:45:01.540579 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:01.540543 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-123986-29893065-dvhsz" event={"ID":"c7ebe1a8-c572-4b18-98d9-083b497aa758","Type":"ContainerStarted","Data":"ffad13c07d1a72c57fa9a0d1ceeee848874b3401c6ec925a5c9ac32c54868862"} Nov 02 01:45:01.540732 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:01.540594 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-123986-29893065-dvhsz" event={"ID":"c7ebe1a8-c572-4b18-98d9-083b497aa758","Type":"ContainerStarted","Data":"2ed4b4053ab25d641b88e0a75d971581e52045f7d7bcee7fcaaa6d9b6388e560"} Nov 02 01:45:01.542252 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:01.542226 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-00f1da-29893065-vkvdj" event={"ID":"5779a226-eaa5-4428-99d1-47e4936591a2","Type":"ContainerStarted","Data":"9cffdc73841c9ac4b4022998f38c8108cf16e54b0fc9ab892c9142971d51098f"} Nov 02 01:45:01.543709 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:01.543684 4427 patch_prober.go:29] interesting pod/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 01:45:01.543827 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:01.543734 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8" podUID="af735a61-695b-4a66-a7cb-d0df06b3b58b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 01:45:01.543827 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:01.543811 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893065-ltmvv" event={"ID":"635a1467-2a08-4b8b-a8c0-b16e2a361888","Type":"ContainerStarted","Data":"e5d68a3b98a0eddeba23af77cfbdff705206f25d56573471377c1a933e2c27c1"} Nov 02 01:45:01.545194 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:01.545173 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9d816-29893065-vvz6j" event={"ID":"bcd48699-8090-4574-934c-b29668788243","Type":"ContainerStarted","Data":"a5e907619671644f7c1ac90274b883e2bcda30974d95593f1c308983b4388085"} Nov 02 01:45:01.546611 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:01.546587 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3c4ea5-29893065-d2hkv" event={"ID":"886fa3e6-9143-4823-903d-7e68d06100ee","Type":"ContainerStarted","Data":"7f5528f3813d7e603ddabf9af36f6de066308994f86dfbc3e28d5a042eb442a2"} Nov 02 01:45:01.564491 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:01.564435 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-123986-29893065-dvhsz" podStartSLOduration=1.56441556 podStartE2EDuration="1.56441556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:45:01.559339066 +0000 UTC m=+1806.180344279" watchObservedRunningTime="2026-11-02 01:45:01.56441556 +0000 UTC m=+1806.185420772" Nov 02 01:45:01.696322 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:45:01.696288 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:45:02.556992 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:02.555947 4427 generic.go:356] "Generic (PLEG): container finished" podID="bcd48699-8090-4574-934c-b29668788243" containerID="0bc65bb2be7b335412b1782d569bb9921f4c3aaf21022fdda0f0c9435a5c4e4a" exitCode=60 Nov 02 01:45:02.556992 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:02.556245 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9d816-29893065-vvz6j" event={"ID":"bcd48699-8090-4574-934c-b29668788243","Type":"ContainerDied","Data":"0bc65bb2be7b335412b1782d569bb9921f4c3aaf21022fdda0f0c9435a5c4e4a"} Nov 02 01:45:02.556992 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:02.556518 4427 scope.go:122] "RemoveContainer" containerID="0bc65bb2be7b335412b1782d569bb9921f4c3aaf21022fdda0f0c9435a5c4e4a" Nov 02 01:45:02.558067 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:02.558043 4427 generic.go:356] "Generic (PLEG): container finished" podID="886fa3e6-9143-4823-903d-7e68d06100ee" containerID="a80ec67f0262c7ed75b64a7ca041ab09f4d6ceb16c04e7c0f4a65aec9b1c5501" exitCode=60 Nov 02 01:45:02.558160 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:02.558129 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3c4ea5-29893065-d2hkv" event={"ID":"886fa3e6-9143-4823-903d-7e68d06100ee","Type":"ContainerDied","Data":"a80ec67f0262c7ed75b64a7ca041ab09f4d6ceb16c04e7c0f4a65aec9b1c5501"} Nov 02 01:45:02.558525 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:02.558439 4427 scope.go:122] "RemoveContainer" containerID="a80ec67f0262c7ed75b64a7ca041ab09f4d6ceb16c04e7c0f4a65aec9b1c5501" Nov 02 01:45:02.560210 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:02.560069 4427 generic.go:356] "Generic (PLEG): container finished" podID="747eccfe-eecb-4cec-99a9-aea7857e1a34" containerID="c15406a16cfc1c6379fff94eabe27ebe7ffd156aed8a66d8fbee5099f2152864" exitCode=60 Nov 02 01:45:02.560210 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:02.560164 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bbb385-29893065-wl5kl" event={"ID":"747eccfe-eecb-4cec-99a9-aea7857e1a34","Type":"ContainerDied","Data":"c15406a16cfc1c6379fff94eabe27ebe7ffd156aed8a66d8fbee5099f2152864"} Nov 02 01:45:02.560454 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:02.560368 4427 scope.go:122] "RemoveContainer" containerID="c15406a16cfc1c6379fff94eabe27ebe7ffd156aed8a66d8fbee5099f2152864" Nov 02 01:45:02.562884 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:02.562662 4427 generic.go:356] "Generic (PLEG): container finished" podID="10bfbdc2-73c3-43b5-9087-bc07be95319e" containerID="22df10b204598ff628f8cc5b573bf9e7776302f508749c7c864bd607091f1633" exitCode=7 Nov 02 01:45:02.562884 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:02.562713 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0f32d6-29893065-5kg4t" event={"ID":"10bfbdc2-73c3-43b5-9087-bc07be95319e","Type":"ContainerDied","Data":"22df10b204598ff628f8cc5b573bf9e7776302f508749c7c864bd607091f1633"} Nov 02 01:45:02.563120 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:02.563036 4427 scope.go:122] "RemoveContainer" containerID="22df10b204598ff628f8cc5b573bf9e7776302f508749c7c864bd607091f1633" Nov 02 01:45:02.566497 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:02.566475 4427 generic.go:356] "Generic (PLEG): container finished" podID="d983a998-461c-4341-aea9-1eb65375bbb7" containerID="f02e582e44efbf4896ea3b7cd88fe0d062a25126c60de245c6e22999fd87e443" exitCode=60 Nov 02 01:45:02.566608 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:02.566545 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e06f2-29893065-4ql7w" event={"ID":"d983a998-461c-4341-aea9-1eb65375bbb7","Type":"ContainerDied","Data":"f02e582e44efbf4896ea3b7cd88fe0d062a25126c60de245c6e22999fd87e443"} Nov 02 01:45:02.566817 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:02.566799 4427 scope.go:122] "RemoveContainer" containerID="f02e582e44efbf4896ea3b7cd88fe0d062a25126c60de245c6e22999fd87e443" Nov 02 01:45:02.568525 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:02.568499 4427 generic.go:356] "Generic (PLEG): container finished" podID="c7ebe1a8-c572-4b18-98d9-083b497aa758" containerID="ffad13c07d1a72c57fa9a0d1ceeee848874b3401c6ec925a5c9ac32c54868862" exitCode=60 Nov 02 01:45:02.568649 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:02.568583 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-123986-29893065-dvhsz" event={"ID":"c7ebe1a8-c572-4b18-98d9-083b497aa758","Type":"ContainerDied","Data":"ffad13c07d1a72c57fa9a0d1ceeee848874b3401c6ec925a5c9ac32c54868862"} Nov 02 01:45:02.568841 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:02.568824 4427 scope.go:122] "RemoveContainer" containerID="ffad13c07d1a72c57fa9a0d1ceeee848874b3401c6ec925a5c9ac32c54868862" Nov 02 01:45:02.582156 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:02.582109 4427 generic.go:356] "Generic (PLEG): container finished" podID="5779a226-eaa5-4428-99d1-47e4936591a2" containerID="7ef8dfb535748779e88eea9508af2a5bf556ed8486121b5d13e2ab81063e899b" exitCode=60 Nov 02 01:45:02.582704 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:02.582532 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-00f1da-29893065-vkvdj" event={"ID":"5779a226-eaa5-4428-99d1-47e4936591a2","Type":"ContainerDied","Data":"7ef8dfb535748779e88eea9508af2a5bf556ed8486121b5d13e2ab81063e899b"} Nov 02 01:45:02.583468 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:02.583155 4427 scope.go:122] "RemoveContainer" containerID="7ef8dfb535748779e88eea9508af2a5bf556ed8486121b5d13e2ab81063e899b" Nov 02 01:45:02.585555 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:02.585526 4427 generic.go:356] "Generic (PLEG): container finished" podID="635a1467-2a08-4b8b-a8c0-b16e2a361888" containerID="a48d295b19e9dbb21807cb63fbc94b6ad41a1266008fbd0b39e0aec7c79d1ac2" exitCode=60 Nov 02 01:45:02.585698 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:02.585583 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893065-ltmvv" event={"ID":"635a1467-2a08-4b8b-a8c0-b16e2a361888","Type":"ContainerDied","Data":"a48d295b19e9dbb21807cb63fbc94b6ad41a1266008fbd0b39e0aec7c79d1ac2"} Nov 02 01:45:02.586013 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:02.585993 4427 scope.go:122] "RemoveContainer" containerID="a48d295b19e9dbb21807cb63fbc94b6ad41a1266008fbd0b39e0aec7c79d1ac2" Nov 02 01:45:03.594853 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:03.594732 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e06f2-29893065-4ql7w" event={"ID":"d983a998-461c-4341-aea9-1eb65375bbb7","Type":"ContainerStarted","Data":"65af2666321da9f96376b9d2addfdd750a831c03963345371127aa05f749ef91"} Nov 02 01:45:03.597233 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:03.597205 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-123986-29893065-dvhsz" event={"ID":"c7ebe1a8-c572-4b18-98d9-083b497aa758","Type":"ContainerStarted","Data":"d126d71477195d8714b425ef3de57080e0717a0d35d204e5a3d9d224f7a6c238"} Nov 02 01:45:03.599602 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:03.599577 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-00f1da-29893065-vkvdj" event={"ID":"5779a226-eaa5-4428-99d1-47e4936591a2","Type":"ContainerStarted","Data":"268b5b0735e3ff50481c3aa866c42c2aad40e4ea8b8516dbe442c52cdeba3e7b"} Nov 02 01:45:03.601672 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:03.601638 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893065-ltmvv" event={"ID":"635a1467-2a08-4b8b-a8c0-b16e2a361888","Type":"ContainerStarted","Data":"6b43c3e5edb45066f089399141f00e5c66636b422e889a1716dd4fce064cf5c1"} Nov 02 01:45:03.603666 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:03.603644 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9d816-29893065-vvz6j" event={"ID":"bcd48699-8090-4574-934c-b29668788243","Type":"ContainerStarted","Data":"2d569355d0d8a976f794fb3315cde993d20d52282d73b571220f475284f311ab"} Nov 02 01:45:03.605751 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:03.605731 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3c4ea5-29893065-d2hkv" event={"ID":"886fa3e6-9143-4823-903d-7e68d06100ee","Type":"ContainerStarted","Data":"e6ebbd7bf01c918acd10d627f57b0dc157e54597ccf49541619167b0ebebe413"} Nov 02 01:45:03.607825 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:03.607806 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bbb385-29893065-wl5kl" event={"ID":"747eccfe-eecb-4cec-99a9-aea7857e1a34","Type":"ContainerStarted","Data":"4acc183591b34d16dfbaa53855f28c500c0acbfb227c3104e5a2a57f117ad44e"} Nov 02 01:45:03.609906 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:03.609887 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0f32d6-29893065-5kg4t" event={"ID":"10bfbdc2-73c3-43b5-9087-bc07be95319e","Type":"ContainerStarted","Data":"2ddce910e8a040b7ba324270b94b58b22551871a1a251846d05b368b2cb445b2"} Nov 02 01:45:03.614881 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:03.614833 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e06f2-29893065-4ql7w" podStartSLOduration=3.614821109 podStartE2EDuration="3.614821109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:45:03.613000319 +0000 UTC m=+1808.234005531" watchObservedRunningTime="2026-11-02 01:45:03.614821109 +0000 UTC m=+1808.235826320" Nov 02 01:45:03.633840 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:03.633792 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-00f1da-29893065-vkvdj" podStartSLOduration=3.633778807 podStartE2EDuration="3.633778807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:45:03.630960141 +0000 UTC m=+1808.251965352" watchObservedRunningTime="2026-11-02 01:45:03.633778807 +0000 UTC m=+1808.254784060" Nov 02 01:45:03.648923 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:03.648870 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9d816-29893065-vvz6j" podStartSLOduration=3.648859158 podStartE2EDuration="3.648859158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:45:03.648251965 +0000 UTC m=+1808.269257177" watchObservedRunningTime="2026-11-02 01:45:03.648859158 +0000 UTC m=+1808.269864369" Nov 02 01:45:03.670538 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:03.670482 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3c4ea5-29893065-d2hkv" podStartSLOduration=3.670467036 podStartE2EDuration="3.670467036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:45:03.666771125 +0000 UTC m=+1808.287776338" watchObservedRunningTime="2026-11-02 01:45:03.670467036 +0000 UTC m=+1808.291472282" Nov 02 01:45:03.685321 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:03.685264 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893065-ltmvv" podStartSLOduration=3.685248524 podStartE2EDuration="3.685248524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:45:03.683890396 +0000 UTC m=+1808.304895608" watchObservedRunningTime="2026-11-02 01:45:03.685248524 +0000 UTC m=+1808.306253779" Nov 02 01:45:03.722724 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:03.722670 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bbb385-29893065-wl5kl" podStartSLOduration=3.72265675 podStartE2EDuration="3.72265675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:45:03.718932788 +0000 UTC m=+1808.339937998" watchObservedRunningTime="2026-11-02 01:45:03.72265675 +0000 UTC m=+1808.343662014" Nov 02 01:45:03.740568 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:03.740510 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0f32d6-29893065-5kg4t" podStartSLOduration=3.740495909 podStartE2EDuration="3.740495909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:45:03.736671458 +0000 UTC m=+1808.357676670" watchObservedRunningTime="2026-11-02 01:45:03.740495909 +0000 UTC m=+1808.361501121" Nov 02 01:45:04.683717 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:04.683625 4427 generic.go:356] "Generic (PLEG): container finished" podID="f3730ee2-6ba0-4dec-b103-214881818132" containerID="a828e3e38efe8596a7047f188634226964ed3fd28db5919d0ae3846282c6744c" exitCode=0 Nov 02 01:45:04.684324 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:04.684294 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7" event={"ID":"f3730ee2-6ba0-4dec-b103-214881818132","Type":"ContainerDied","Data":"a828e3e38efe8596a7047f188634226964ed3fd28db5919d0ae3846282c6744c"} Nov 02 01:45:04.689543 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:04.687455 4427 generic.go:356] "Generic (PLEG): container finished" podID="747eccfe-eecb-4cec-99a9-aea7857e1a34" containerID="4acc183591b34d16dfbaa53855f28c500c0acbfb227c3104e5a2a57f117ad44e" exitCode=60 Nov 02 01:45:04.689798 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:04.689773 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bbb385-29893065-wl5kl" event={"ID":"747eccfe-eecb-4cec-99a9-aea7857e1a34","Type":"ContainerDied","Data":"4acc183591b34d16dfbaa53855f28c500c0acbfb227c3104e5a2a57f117ad44e"} Nov 02 01:45:04.689957 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:04.689944 4427 scope.go:122] "RemoveContainer" containerID="c15406a16cfc1c6379fff94eabe27ebe7ffd156aed8a66d8fbee5099f2152864" Nov 02 01:45:04.690666 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:04.690629 4427 scope.go:122] "RemoveContainer" containerID="4acc183591b34d16dfbaa53855f28c500c0acbfb227c3104e5a2a57f117ad44e" Nov 02 01:45:04.691358 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:45:04.691335 4427 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-bbb385-29893065-wl5kl_odh-ai-gateway-infra(747eccfe-eecb-4cec-99a9-aea7857e1a34)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bbb385-29893065-wl5kl" podUID="747eccfe-eecb-4cec-99a9-aea7857e1a34" Nov 02 01:45:04.704296 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:04.704247 4427 generic.go:356] "Generic (PLEG): container finished" podID="10bfbdc2-73c3-43b5-9087-bc07be95319e" containerID="2ddce910e8a040b7ba324270b94b58b22551871a1a251846d05b368b2cb445b2" exitCode=7 Nov 02 01:45:04.704569 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:04.704545 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0f32d6-29893065-5kg4t" event={"ID":"10bfbdc2-73c3-43b5-9087-bc07be95319e","Type":"ContainerDied","Data":"2ddce910e8a040b7ba324270b94b58b22551871a1a251846d05b368b2cb445b2"} Nov 02 01:45:04.705659 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:04.705638 4427 scope.go:122] "RemoveContainer" containerID="2ddce910e8a040b7ba324270b94b58b22551871a1a251846d05b368b2cb445b2" Nov 02 01:45:04.706272 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:45:04.706231 4427 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-0f32d6-29893065-5kg4t_odh-ai-gateway-infra(10bfbdc2-73c3-43b5-9087-bc07be95319e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0f32d6-29893065-5kg4t" podUID="10bfbdc2-73c3-43b5-9087-bc07be95319e" Nov 02 01:45:04.717993 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:04.717962 4427 generic.go:356] "Generic (PLEG): container finished" podID="d983a998-461c-4341-aea9-1eb65375bbb7" containerID="65af2666321da9f96376b9d2addfdd750a831c03963345371127aa05f749ef91" exitCode=60 Nov 02 01:45:04.718259 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:04.718230 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e06f2-29893065-4ql7w" event={"ID":"d983a998-461c-4341-aea9-1eb65375bbb7","Type":"ContainerDied","Data":"65af2666321da9f96376b9d2addfdd750a831c03963345371127aa05f749ef91"} Nov 02 01:45:04.718871 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:04.718851 4427 scope.go:122] "RemoveContainer" containerID="65af2666321da9f96376b9d2addfdd750a831c03963345371127aa05f749ef91" Nov 02 01:45:04.719471 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:45:04.719446 4427 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-0e06f2-29893065-4ql7w_odh-ai-gateway-infra(d983a998-461c-4341-aea9-1eb65375bbb7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e06f2-29893065-4ql7w" podUID="d983a998-461c-4341-aea9-1eb65375bbb7" Nov 02 01:45:04.722162 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:04.722121 4427 generic.go:356] "Generic (PLEG): container finished" podID="af735a61-695b-4a66-a7cb-d0df06b3b58b" containerID="40e8a42407d0d5421dd8d43594a03c7f6cdcf21528adc613f5c28d4bf54a2c65" exitCode=0 Nov 02 01:45:04.722552 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:04.722519 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8" event={"ID":"af735a61-695b-4a66-a7cb-d0df06b3b58b","Type":"ContainerDied","Data":"40e8a42407d0d5421dd8d43594a03c7f6cdcf21528adc613f5c28d4bf54a2c65"} Nov 02 01:45:04.729923 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:04.729888 4427 generic.go:356] "Generic (PLEG): container finished" podID="c7ebe1a8-c572-4b18-98d9-083b497aa758" containerID="d126d71477195d8714b425ef3de57080e0717a0d35d204e5a3d9d224f7a6c238" exitCode=60 Nov 02 01:45:04.730170 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:04.730149 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-123986-29893065-dvhsz" event={"ID":"c7ebe1a8-c572-4b18-98d9-083b497aa758","Type":"ContainerDied","Data":"d126d71477195d8714b425ef3de57080e0717a0d35d204e5a3d9d224f7a6c238"} Nov 02 01:45:04.730821 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:04.730799 4427 scope.go:122] "RemoveContainer" containerID="d126d71477195d8714b425ef3de57080e0717a0d35d204e5a3d9d224f7a6c238" Nov 02 01:45:04.734422 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:04.734390 4427 generic.go:356] "Generic (PLEG): container finished" podID="5779a226-eaa5-4428-99d1-47e4936591a2" containerID="268b5b0735e3ff50481c3aa866c42c2aad40e4ea8b8516dbe442c52cdeba3e7b" exitCode=60 Nov 02 01:45:04.734705 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:04.734682 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-00f1da-29893065-vkvdj" event={"ID":"5779a226-eaa5-4428-99d1-47e4936591a2","Type":"ContainerDied","Data":"268b5b0735e3ff50481c3aa866c42c2aad40e4ea8b8516dbe442c52cdeba3e7b"} Nov 02 01:45:04.735362 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:04.735299 4427 scope.go:122] "RemoveContainer" containerID="268b5b0735e3ff50481c3aa866c42c2aad40e4ea8b8516dbe442c52cdeba3e7b" Nov 02 01:45:04.736028 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:45:04.736003 4427 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-00f1da-29893065-vkvdj_odh-ai-gateway-infra(5779a226-eaa5-4428-99d1-47e4936591a2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-00f1da-29893065-vkvdj" podUID="5779a226-eaa5-4428-99d1-47e4936591a2" Nov 02 01:45:04.740428 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:04.739622 4427 generic.go:356] "Generic (PLEG): container finished" podID="635a1467-2a08-4b8b-a8c0-b16e2a361888" containerID="6b43c3e5edb45066f089399141f00e5c66636b422e889a1716dd4fce064cf5c1" exitCode=60 Nov 02 01:45:04.740428 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:04.739709 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893065-ltmvv" event={"ID":"635a1467-2a08-4b8b-a8c0-b16e2a361888","Type":"ContainerDied","Data":"6b43c3e5edb45066f089399141f00e5c66636b422e889a1716dd4fce064cf5c1"} Nov 02 01:45:04.740428 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:04.740181 4427 scope.go:122] "RemoveContainer" containerID="6b43c3e5edb45066f089399141f00e5c66636b422e889a1716dd4fce064cf5c1" Nov 02 01:45:04.741259 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:45:04.740896 4427 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893065-ltmvv_odh-ai-gateway-infra(635a1467-2a08-4b8b-a8c0-b16e2a361888)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893065-ltmvv" podUID="635a1467-2a08-4b8b-a8c0-b16e2a361888" Nov 02 01:45:04.744261 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:04.743251 4427 generic.go:356] "Generic (PLEG): container finished" podID="bcd48699-8090-4574-934c-b29668788243" containerID="2d569355d0d8a976f794fb3315cde993d20d52282d73b571220f475284f311ab" exitCode=60 Nov 02 01:45:04.744261 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:04.743323 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9d816-29893065-vvz6j" event={"ID":"bcd48699-8090-4574-934c-b29668788243","Type":"ContainerDied","Data":"2d569355d0d8a976f794fb3315cde993d20d52282d73b571220f475284f311ab"} Nov 02 01:45:04.744261 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:04.743793 4427 scope.go:122] "RemoveContainer" containerID="2d569355d0d8a976f794fb3315cde993d20d52282d73b571220f475284f311ab" Nov 02 01:45:04.744261 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:45:04.744184 4427 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-b9d816-29893065-vvz6j_odh-ai-gateway-infra(bcd48699-8090-4574-934c-b29668788243)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9d816-29893065-vvz6j" podUID="bcd48699-8090-4574-934c-b29668788243" Nov 02 01:45:04.753173 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:04.747182 4427 generic.go:356] "Generic (PLEG): container finished" podID="886fa3e6-9143-4823-903d-7e68d06100ee" containerID="e6ebbd7bf01c918acd10d627f57b0dc157e54597ccf49541619167b0ebebe413" exitCode=60 Nov 02 01:45:04.753173 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:04.747234 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3c4ea5-29893065-d2hkv" event={"ID":"886fa3e6-9143-4823-903d-7e68d06100ee","Type":"ContainerDied","Data":"e6ebbd7bf01c918acd10d627f57b0dc157e54597ccf49541619167b0ebebe413"} Nov 02 01:45:04.753173 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:04.747748 4427 scope.go:122] "RemoveContainer" containerID="e6ebbd7bf01c918acd10d627f57b0dc157e54597ccf49541619167b0ebebe413" Nov 02 01:45:04.753173 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:45:04.748138 4427 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-3c4ea5-29893065-d2hkv_odh-ai-gateway-infra(886fa3e6-9143-4823-903d-7e68d06100ee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3c4ea5-29893065-d2hkv" podUID="886fa3e6-9143-4823-903d-7e68d06100ee" Nov 02 01:45:04.779310 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:45:04.779227 4427 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-123986-29893065-dvhsz_odh-ai-gateway-infra(c7ebe1a8-c572-4b18-98d9-083b497aa758)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-123986-29893065-dvhsz" podUID="c7ebe1a8-c572-4b18-98d9-083b497aa758" Nov 02 01:45:04.865006 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:04.864509 4427 scope.go:122] "RemoveContainer" containerID="22df10b204598ff628f8cc5b573bf9e7776302f508749c7c864bd607091f1633" Nov 02 01:45:04.872356 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:04.872323 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8" Nov 02 01:45:04.884507 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:04.884479 4427 scope.go:122] "RemoveContainer" containerID="f02e582e44efbf4896ea3b7cd88fe0d062a25126c60de245c6e22999fd87e443" Nov 02 01:45:04.902081 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:04.902055 4427 scope.go:122] "RemoveContainer" containerID="ffad13c07d1a72c57fa9a0d1ceeee848874b3401c6ec925a5c9ac32c54868862" Nov 02 01:45:04.921563 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:04.921525 4427 scope.go:122] "RemoveContainer" containerID="7ef8dfb535748779e88eea9508af2a5bf556ed8486121b5d13e2ab81063e899b" Nov 02 01:45:04.939525 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:04.939353 4427 scope.go:122] "RemoveContainer" containerID="a48d295b19e9dbb21807cb63fbc94b6ad41a1266008fbd0b39e0aec7c79d1ac2" Nov 02 01:45:04.957297 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:04.957276 4427 scope.go:122] "RemoveContainer" containerID="0bc65bb2be7b335412b1782d569bb9921f4c3aaf21022fdda0f0c9435a5c4e4a" Nov 02 01:45:04.975233 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:04.975202 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af735a61-695b-4a66-a7cb-d0df06b3b58b-kube-api-access-wf5cg\" (UniqueName: \"kubernetes.io/projected/af735a61-695b-4a66-a7cb-d0df06b3b58b-kube-api-access-wf5cg\") pod \"af735a61-695b-4a66-a7cb-d0df06b3b58b\" (UID: \"af735a61-695b-4a66-a7cb-d0df06b3b58b\") " Nov 02 01:45:04.975609 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:04.975563 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af735a61-695b-4a66-a7cb-d0df06b3b58b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af735a61-695b-4a66-a7cb-d0df06b3b58b-workload-socket\") pod \"af735a61-695b-4a66-a7cb-d0df06b3b58b\" (UID: \"af735a61-695b-4a66-a7cb-d0df06b3b58b\") " Nov 02 01:45:04.975901 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:04.975857 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af735a61-695b-4a66-a7cb-d0df06b3b58b-workload-socket" pod "af735a61-695b-4a66-a7cb-d0df06b3b58b" (UID: "af735a61-695b-4a66-a7cb-d0df06b3b58b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:04.976099 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:04.976080 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af735a61-695b-4a66-a7cb-d0df06b3b58b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af735a61-695b-4a66-a7cb-d0df06b3b58b-istio-data\") pod \"af735a61-695b-4a66-a7cb-d0df06b3b58b\" (UID: \"af735a61-695b-4a66-a7cb-d0df06b3b58b\") " Nov 02 01:45:04.976452 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:04.976434 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af735a61-695b-4a66-a7cb-d0df06b3b58b-istio-token\" (UniqueName: \"kubernetes.io/projected/af735a61-695b-4a66-a7cb-d0df06b3b58b-istio-token\") pod \"af735a61-695b-4a66-a7cb-d0df06b3b58b\" (UID: \"af735a61-695b-4a66-a7cb-d0df06b3b58b\") " Nov 02 01:45:04.977220 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:04.977200 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af735a61-695b-4a66-a7cb-d0df06b3b58b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af735a61-695b-4a66-a7cb-d0df06b3b58b-istio-podinfo\") pod \"af735a61-695b-4a66-a7cb-d0df06b3b58b\" (UID: \"af735a61-695b-4a66-a7cb-d0df06b3b58b\") " Nov 02 01:45:04.977714 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:04.977697 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af735a61-695b-4a66-a7cb-d0df06b3b58b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af735a61-695b-4a66-a7cb-d0df06b3b58b-istio-envoy\") pod \"af735a61-695b-4a66-a7cb-d0df06b3b58b\" (UID: \"af735a61-695b-4a66-a7cb-d0df06b3b58b\") " Nov 02 01:45:04.978160 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:04.978142 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af735a61-695b-4a66-a7cb-d0df06b3b58b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af735a61-695b-4a66-a7cb-d0df06b3b58b-workload-certs\") pod \"af735a61-695b-4a66-a7cb-d0df06b3b58b\" (UID: \"af735a61-695b-4a66-a7cb-d0df06b3b58b\") " Nov 02 01:45:04.978666 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:04.978649 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af735a61-695b-4a66-a7cb-d0df06b3b58b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af735a61-695b-4a66-a7cb-d0df06b3b58b-credential-socket\") pod \"af735a61-695b-4a66-a7cb-d0df06b3b58b\" (UID: \"af735a61-695b-4a66-a7cb-d0df06b3b58b\") " Nov 02 01:45:04.979179 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:04.979161 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af735a61-695b-4a66-a7cb-d0df06b3b58b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af735a61-695b-4a66-a7cb-d0df06b3b58b-istiod-ca-cert\") pod \"af735a61-695b-4a66-a7cb-d0df06b3b58b\" (UID: \"af735a61-695b-4a66-a7cb-d0df06b3b58b\") " Nov 02 01:45:04.980476 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:04.980458 4427 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af735a61-695b-4a66-a7cb-d0df06b3b58b-workload-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:04.980600 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:04.976357 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af735a61-695b-4a66-a7cb-d0df06b3b58b-istio-data" pod "af735a61-695b-4a66-a7cb-d0df06b3b58b" (UID: "af735a61-695b-4a66-a7cb-d0df06b3b58b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:04.980684 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:04.978503 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af735a61-695b-4a66-a7cb-d0df06b3b58b-workload-certs" pod "af735a61-695b-4a66-a7cb-d0df06b3b58b" (UID: "af735a61-695b-4a66-a7cb-d0df06b3b58b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:04.980772 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:04.978981 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af735a61-695b-4a66-a7cb-d0df06b3b58b-credential-socket" pod "af735a61-695b-4a66-a7cb-d0df06b3b58b" (UID: "af735a61-695b-4a66-a7cb-d0df06b3b58b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:04.980843 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:04.979574 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af735a61-695b-4a66-a7cb-d0df06b3b58b-istiod-ca-cert" pod "af735a61-695b-4a66-a7cb-d0df06b3b58b" (UID: "af735a61-695b-4a66-a7cb-d0df06b3b58b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:45:04.986904 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:04.986871 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af735a61-695b-4a66-a7cb-d0df06b3b58b-kube-api-access-wf5cg" pod "af735a61-695b-4a66-a7cb-d0df06b3b58b" (UID: "af735a61-695b-4a66-a7cb-d0df06b3b58b"). InnerVolumeSpecName "kube-api-access-wf5cg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:45:04.990554 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:04.990522 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af735a61-695b-4a66-a7cb-d0df06b3b58b-istio-envoy" pod "af735a61-695b-4a66-a7cb-d0df06b3b58b" (UID: "af735a61-695b-4a66-a7cb-d0df06b3b58b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:04.992018 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:04.990537 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af735a61-695b-4a66-a7cb-d0df06b3b58b-istio-podinfo" pod "af735a61-695b-4a66-a7cb-d0df06b3b58b" (UID: "af735a61-695b-4a66-a7cb-d0df06b3b58b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:45:04.992179 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:04.990593 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af735a61-695b-4a66-a7cb-d0df06b3b58b-istio-token" pod "af735a61-695b-4a66-a7cb-d0df06b3b58b" (UID: "af735a61-695b-4a66-a7cb-d0df06b3b58b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:45:04.994575 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:04.994490 4427 scope.go:122] "RemoveContainer" containerID="a80ec67f0262c7ed75b64a7ca041ab09f4d6ceb16c04e7c0f4a65aec9b1c5501" Nov 02 01:45:04.996096 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:04.996076 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7" Nov 02 01:45:05.083036 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:05.082978 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f3730ee2-6ba0-4dec-b103-214881818132-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3730ee2-6ba0-4dec-b103-214881818132-istio-podinfo\") pod \"f3730ee2-6ba0-4dec-b103-214881818132\" (UID: \"f3730ee2-6ba0-4dec-b103-214881818132\") " Nov 02 01:45:05.083250 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:05.083172 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3730ee2-6ba0-4dec-b103-214881818132-kube-api-access-qxq28\" (UniqueName: \"kubernetes.io/projected/f3730ee2-6ba0-4dec-b103-214881818132-kube-api-access-qxq28\") pod \"f3730ee2-6ba0-4dec-b103-214881818132\" (UID: \"f3730ee2-6ba0-4dec-b103-214881818132\") " Nov 02 01:45:05.083416 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:05.083353 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3730ee2-6ba0-4dec-b103-214881818132-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3730ee2-6ba0-4dec-b103-214881818132-workload-certs\") pod \"f3730ee2-6ba0-4dec-b103-214881818132\" (UID: \"f3730ee2-6ba0-4dec-b103-214881818132\") " Nov 02 01:45:05.083505 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:05.083421 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3730ee2-6ba0-4dec-b103-214881818132-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3730ee2-6ba0-4dec-b103-214881818132-istio-envoy\") pod \"f3730ee2-6ba0-4dec-b103-214881818132\" (UID: \"f3730ee2-6ba0-4dec-b103-214881818132\") " Nov 02 01:45:05.083567 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:05.083492 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3730ee2-6ba0-4dec-b103-214881818132-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3730ee2-6ba0-4dec-b103-214881818132-istiod-ca-cert\") pod \"f3730ee2-6ba0-4dec-b103-214881818132\" (UID: \"f3730ee2-6ba0-4dec-b103-214881818132\") " Nov 02 01:45:05.083701 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:05.083670 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3730ee2-6ba0-4dec-b103-214881818132-istio-token\" (UniqueName: \"kubernetes.io/projected/f3730ee2-6ba0-4dec-b103-214881818132-istio-token\") pod \"f3730ee2-6ba0-4dec-b103-214881818132\" (UID: \"f3730ee2-6ba0-4dec-b103-214881818132\") " Nov 02 01:45:05.083785 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:05.083760 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3730ee2-6ba0-4dec-b103-214881818132-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3730ee2-6ba0-4dec-b103-214881818132-workload-socket\") pod \"f3730ee2-6ba0-4dec-b103-214881818132\" (UID: \"f3730ee2-6ba0-4dec-b103-214881818132\") " Nov 02 01:45:05.083954 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:05.083936 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3730ee2-6ba0-4dec-b103-214881818132-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3730ee2-6ba0-4dec-b103-214881818132-istio-data\") pod \"f3730ee2-6ba0-4dec-b103-214881818132\" (UID: \"f3730ee2-6ba0-4dec-b103-214881818132\") " Nov 02 01:45:05.084031 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:05.083971 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3730ee2-6ba0-4dec-b103-214881818132-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3730ee2-6ba0-4dec-b103-214881818132-credential-socket\") pod \"f3730ee2-6ba0-4dec-b103-214881818132\" (UID: \"f3730ee2-6ba0-4dec-b103-214881818132\") " Nov 02 01:45:05.084468 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:05.084439 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3730ee2-6ba0-4dec-b103-214881818132-istiod-ca-cert" pod "f3730ee2-6ba0-4dec-b103-214881818132" (UID: "f3730ee2-6ba0-4dec-b103-214881818132"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:45:05.084684 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:05.084661 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3730ee2-6ba0-4dec-b103-214881818132-credential-socket" pod "f3730ee2-6ba0-4dec-b103-214881818132" (UID: "f3730ee2-6ba0-4dec-b103-214881818132"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:05.084804 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:05.084790 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf5cg\" (UniqueName: \"kubernetes.io/projected/af735a61-695b-4a66-a7cb-d0df06b3b58b-kube-api-access-wf5cg\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:05.084872 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:05.084807 4427 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af735a61-695b-4a66-a7cb-d0df06b3b58b-istio-data\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:05.084872 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:05.084816 4427 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af735a61-695b-4a66-a7cb-d0df06b3b58b-istio-token\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:05.084872 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:05.084828 4427 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3730ee2-6ba0-4dec-b103-214881818132-istiod-ca-cert\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:05.084872 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:05.084836 4427 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af735a61-695b-4a66-a7cb-d0df06b3b58b-istio-podinfo\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:05.084872 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:05.084869 4427 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af735a61-695b-4a66-a7cb-d0df06b3b58b-istio-envoy\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:05.085140 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:05.084879 4427 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af735a61-695b-4a66-a7cb-d0df06b3b58b-workload-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:05.085140 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:05.084887 4427 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af735a61-695b-4a66-a7cb-d0df06b3b58b-credential-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:05.085140 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:05.084896 4427 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3730ee2-6ba0-4dec-b103-214881818132-credential-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:05.085140 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:05.084898 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3730ee2-6ba0-4dec-b103-214881818132-workload-certs" pod "f3730ee2-6ba0-4dec-b103-214881818132" (UID: "f3730ee2-6ba0-4dec-b103-214881818132"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:05.085140 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:05.084905 4427 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af735a61-695b-4a66-a7cb-d0df06b3b58b-istiod-ca-cert\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:05.085719 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:05.085482 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3730ee2-6ba0-4dec-b103-214881818132-istio-data" pod "f3730ee2-6ba0-4dec-b103-214881818132" (UID: "f3730ee2-6ba0-4dec-b103-214881818132"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:05.085719 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:05.085550 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3730ee2-6ba0-4dec-b103-214881818132-workload-socket" pod "f3730ee2-6ba0-4dec-b103-214881818132" (UID: "f3730ee2-6ba0-4dec-b103-214881818132"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:05.088284 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:05.087579 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3730ee2-6ba0-4dec-b103-214881818132-kube-api-access-qxq28" pod "f3730ee2-6ba0-4dec-b103-214881818132" (UID: "f3730ee2-6ba0-4dec-b103-214881818132"). InnerVolumeSpecName "kube-api-access-qxq28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:45:05.089606 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:05.089574 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f3730ee2-6ba0-4dec-b103-214881818132-istio-podinfo" pod "f3730ee2-6ba0-4dec-b103-214881818132" (UID: "f3730ee2-6ba0-4dec-b103-214881818132"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:45:05.089878 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:05.089856 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3730ee2-6ba0-4dec-b103-214881818132-istio-token" pod "f3730ee2-6ba0-4dec-b103-214881818132" (UID: "f3730ee2-6ba0-4dec-b103-214881818132"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:45:05.090567 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:05.090543 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3730ee2-6ba0-4dec-b103-214881818132-istio-envoy" pod "f3730ee2-6ba0-4dec-b103-214881818132" (UID: "f3730ee2-6ba0-4dec-b103-214881818132"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:05.186208 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:05.186158 4427 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3730ee2-6ba0-4dec-b103-214881818132-istio-token\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:05.186208 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:05.186200 4427 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3730ee2-6ba0-4dec-b103-214881818132-workload-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:05.186208 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:05.186216 4427 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3730ee2-6ba0-4dec-b103-214881818132-istio-data\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:05.186487 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:05.186229 4427 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3730ee2-6ba0-4dec-b103-214881818132-istio-podinfo\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:05.186487 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:05.186246 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxq28\" (UniqueName: \"kubernetes.io/projected/f3730ee2-6ba0-4dec-b103-214881818132-kube-api-access-qxq28\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:05.186487 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:05.186258 4427 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3730ee2-6ba0-4dec-b103-214881818132-workload-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:05.186487 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:05.186269 4427 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3730ee2-6ba0-4dec-b103-214881818132-istio-envoy\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:05.492301 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:05.492264 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d4a0-6595f44dff-m47jt" Nov 02 01:45:05.758778 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:05.758696 4427 scope.go:122] "RemoveContainer" containerID="d126d71477195d8714b425ef3de57080e0717a0d35d204e5a3d9d224f7a6c238" Nov 02 01:45:05.759243 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:45:05.759118 4427 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-123986-29893065-dvhsz_odh-ai-gateway-infra(c7ebe1a8-c572-4b18-98d9-083b497aa758)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-123986-29893065-dvhsz" podUID="c7ebe1a8-c572-4b18-98d9-083b497aa758" Nov 02 01:45:05.760512 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:05.760478 4427 scope.go:122] "RemoveContainer" containerID="268b5b0735e3ff50481c3aa866c42c2aad40e4ea8b8516dbe442c52cdeba3e7b" Nov 02 01:45:05.760875 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:45:05.760849 4427 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-00f1da-29893065-vkvdj_odh-ai-gateway-infra(5779a226-eaa5-4428-99d1-47e4936591a2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-00f1da-29893065-vkvdj" podUID="5779a226-eaa5-4428-99d1-47e4936591a2" Nov 02 01:45:05.762354 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:05.762330 4427 scope.go:122] "RemoveContainer" containerID="6b43c3e5edb45066f089399141f00e5c66636b422e889a1716dd4fce064cf5c1" Nov 02 01:45:05.762761 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:45:05.762735 4427 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893065-ltmvv_odh-ai-gateway-infra(635a1467-2a08-4b8b-a8c0-b16e2a361888)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893065-ltmvv" podUID="635a1467-2a08-4b8b-a8c0-b16e2a361888" Nov 02 01:45:05.764274 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:05.764251 4427 scope.go:122] "RemoveContainer" containerID="2d569355d0d8a976f794fb3315cde993d20d52282d73b571220f475284f311ab" Nov 02 01:45:05.764624 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:45:05.764599 4427 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-b9d816-29893065-vvz6j_odh-ai-gateway-infra(bcd48699-8090-4574-934c-b29668788243)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9d816-29893065-vvz6j" podUID="bcd48699-8090-4574-934c-b29668788243" Nov 02 01:45:05.766118 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:05.766099 4427 scope.go:122] "RemoveContainer" containerID="e6ebbd7bf01c918acd10d627f57b0dc157e54597ccf49541619167b0ebebe413" Nov 02 01:45:05.766460 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:45:05.766437 4427 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-3c4ea5-29893065-d2hkv_odh-ai-gateway-infra(886fa3e6-9143-4823-903d-7e68d06100ee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3c4ea5-29893065-d2hkv" podUID="886fa3e6-9143-4823-903d-7e68d06100ee" Nov 02 01:45:05.767324 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:05.767303 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7" event={"ID":"f3730ee2-6ba0-4dec-b103-214881818132","Type":"ContainerDied","Data":"ef4c3729061f1c2c96c8eac6d465f5f17b0388216823b2b50c66464b9148cdb2"} Nov 02 01:45:05.767324 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:05.767317 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7" Nov 02 01:45:05.767525 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:05.767365 4427 scope.go:122] "RemoveContainer" containerID="a828e3e38efe8596a7047f188634226964ed3fd28db5919d0ae3846282c6744c" Nov 02 01:45:05.769560 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:05.769533 4427 scope.go:122] "RemoveContainer" containerID="4acc183591b34d16dfbaa53855f28c500c0acbfb227c3104e5a2a57f117ad44e" Nov 02 01:45:05.770159 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:45:05.769823 4427 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-bbb385-29893065-wl5kl_odh-ai-gateway-infra(747eccfe-eecb-4cec-99a9-aea7857e1a34)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bbb385-29893065-wl5kl" podUID="747eccfe-eecb-4cec-99a9-aea7857e1a34" Nov 02 01:45:05.772554 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:05.772532 4427 scope.go:122] "RemoveContainer" containerID="2ddce910e8a040b7ba324270b94b58b22551871a1a251846d05b368b2cb445b2" Nov 02 01:45:05.772849 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:45:05.772831 4427 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-0f32d6-29893065-5kg4t_odh-ai-gateway-infra(10bfbdc2-73c3-43b5-9087-bc07be95319e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0f32d6-29893065-5kg4t" podUID="10bfbdc2-73c3-43b5-9087-bc07be95319e" Nov 02 01:45:05.775412 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:05.775318 4427 scope.go:122] "RemoveContainer" containerID="65af2666321da9f96376b9d2addfdd750a831c03963345371127aa05f749ef91" Nov 02 01:45:05.775635 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:45:05.775612 4427 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-0e06f2-29893065-4ql7w_odh-ai-gateway-infra(d983a998-461c-4341-aea9-1eb65375bbb7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e06f2-29893065-4ql7w" podUID="d983a998-461c-4341-aea9-1eb65375bbb7" Nov 02 01:45:05.776662 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:05.776640 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8" event={"ID":"af735a61-695b-4a66-a7cb-d0df06b3b58b","Type":"ContainerDied","Data":"b4d7f9852bb0b2fb7dc983f0952de09e4b4b4d71d4aff7f2ea71ea4543c65936"} Nov 02 01:45:05.776759 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:05.776674 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8" Nov 02 01:45:05.781135 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:05.781109 4427 scope.go:122] "RemoveContainer" containerID="40e8a42407d0d5421dd8d43594a03c7f6cdcf21528adc613f5c28d4bf54a2c65" Nov 02 01:45:06.095479 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:06.095189 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7"] Nov 02 01:45:06.112404 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:06.102937 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-p5lk7"] Nov 02 01:45:06.112404 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:06.106804 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8"] Nov 02 01:45:06.126342 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:06.118775 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b6ec07-openshift-default-ddc69455f-c6wr8"] Nov 02 01:45:07.538130 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:07.538096 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af735a61-695b-4a66-a7cb-d0df06b3b58b" path="/var/lib/kubelet/pods/af735a61-695b-4a66-a7cb-d0df06b3b58b/volumes" Nov 02 01:45:07.538797 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:07.538780 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3730ee2-6ba0-4dec-b103-214881818132" path="/var/lib/kubelet/pods/f3730ee2-6ba0-4dec-b103-214881818132/volumes" Nov 02 01:45:14.526589 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:14.526554 4427 scope.go:122] "RemoveContainer" containerID="6b43c3e5edb45066f089399141f00e5c66636b422e889a1716dd4fce064cf5c1" Nov 02 01:45:15.535682 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:15.535646 4427 scope.go:122] "RemoveContainer" containerID="268b5b0735e3ff50481c3aa866c42c2aad40e4ea8b8516dbe442c52cdeba3e7b" Nov 02 01:45:15.536202 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:15.535932 4427 scope.go:122] "RemoveContainer" containerID="d126d71477195d8714b425ef3de57080e0717a0d35d204e5a3d9d224f7a6c238" Nov 02 01:45:15.537326 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:15.536665 4427 scope.go:122] "RemoveContainer" containerID="2d569355d0d8a976f794fb3315cde993d20d52282d73b571220f475284f311ab" Nov 02 01:45:15.960197 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:15.960157 4427 generic.go:356] "Generic (PLEG): container finished" podID="635a1467-2a08-4b8b-a8c0-b16e2a361888" containerID="4efbe14aca4d71912deb75c5851640b908c912b3f1293b99de2c1316ff19ba26" exitCode=60 Nov 02 01:45:15.960348 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:15.960209 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893065-ltmvv" event={"ID":"635a1467-2a08-4b8b-a8c0-b16e2a361888","Type":"ContainerDied","Data":"4efbe14aca4d71912deb75c5851640b908c912b3f1293b99de2c1316ff19ba26"} Nov 02 01:45:15.960348 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:15.960264 4427 scope.go:122] "RemoveContainer" containerID="6b43c3e5edb45066f089399141f00e5c66636b422e889a1716dd4fce064cf5c1" Nov 02 01:45:15.960760 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:15.960726 4427 scope.go:122] "RemoveContainer" containerID="4efbe14aca4d71912deb75c5851640b908c912b3f1293b99de2c1316ff19ba26" Nov 02 01:45:15.961209 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:45:15.961179 4427 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893065-ltmvv_odh-ai-gateway-infra(635a1467-2a08-4b8b-a8c0-b16e2a361888)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893065-ltmvv" podUID="635a1467-2a08-4b8b-a8c0-b16e2a361888" Nov 02 01:45:16.527220 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:16.527193 4427 scope.go:122] "RemoveContainer" containerID="65af2666321da9f96376b9d2addfdd750a831c03963345371127aa05f749ef91" Nov 02 01:45:16.970275 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:16.970244 4427 generic.go:356] "Generic (PLEG): container finished" podID="c7ebe1a8-c572-4b18-98d9-083b497aa758" containerID="a1e9e78ca81b59ebdd95b5980f3be099da3590aee7f0d02449196288e6b2cb49" exitCode=60 Nov 02 01:45:16.970773 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:16.970312 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-123986-29893065-dvhsz" event={"ID":"c7ebe1a8-c572-4b18-98d9-083b497aa758","Type":"ContainerDied","Data":"a1e9e78ca81b59ebdd95b5980f3be099da3590aee7f0d02449196288e6b2cb49"} Nov 02 01:45:16.970773 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:16.970364 4427 scope.go:122] "RemoveContainer" containerID="d126d71477195d8714b425ef3de57080e0717a0d35d204e5a3d9d224f7a6c238" Nov 02 01:45:16.970905 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:16.970887 4427 scope.go:122] "RemoveContainer" containerID="a1e9e78ca81b59ebdd95b5980f3be099da3590aee7f0d02449196288e6b2cb49" Nov 02 01:45:16.971466 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:45:16.971440 4427 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-123986-29893065-dvhsz_odh-ai-gateway-infra(c7ebe1a8-c572-4b18-98d9-083b497aa758)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-123986-29893065-dvhsz" podUID="c7ebe1a8-c572-4b18-98d9-083b497aa758" Nov 02 01:45:16.973025 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:16.973002 4427 generic.go:356] "Generic (PLEG): container finished" podID="5779a226-eaa5-4428-99d1-47e4936591a2" containerID="9bb38d2157dddb649b4cacb4610ec2845df8795070d438a619635c16f4f670c3" exitCode=60 Nov 02 01:45:16.973112 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:16.973070 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-00f1da-29893065-vkvdj" event={"ID":"5779a226-eaa5-4428-99d1-47e4936591a2","Type":"ContainerDied","Data":"9bb38d2157dddb649b4cacb4610ec2845df8795070d438a619635c16f4f670c3"} Nov 02 01:45:16.973450 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:16.973427 4427 scope.go:122] "RemoveContainer" containerID="9bb38d2157dddb649b4cacb4610ec2845df8795070d438a619635c16f4f670c3" Nov 02 01:45:16.973745 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:45:16.973719 4427 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-00f1da-29893065-vkvdj_odh-ai-gateway-infra(5779a226-eaa5-4428-99d1-47e4936591a2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-00f1da-29893065-vkvdj" podUID="5779a226-eaa5-4428-99d1-47e4936591a2" Nov 02 01:45:16.977857 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:16.977835 4427 generic.go:356] "Generic (PLEG): container finished" podID="bcd48699-8090-4574-934c-b29668788243" containerID="2d20a65009436527a20d88f10ac9a1cb6f51de8c535e9c9680021020e5cf480c" exitCode=60 Nov 02 01:45:16.977963 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:16.977914 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9d816-29893065-vvz6j" event={"ID":"bcd48699-8090-4574-934c-b29668788243","Type":"ContainerDied","Data":"2d20a65009436527a20d88f10ac9a1cb6f51de8c535e9c9680021020e5cf480c"} Nov 02 01:45:16.978519 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:16.978498 4427 scope.go:122] "RemoveContainer" containerID="2d20a65009436527a20d88f10ac9a1cb6f51de8c535e9c9680021020e5cf480c" Nov 02 01:45:16.978887 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:45:16.978860 4427 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-b9d816-29893065-vvz6j_odh-ai-gateway-infra(bcd48699-8090-4574-934c-b29668788243)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9d816-29893065-vvz6j" podUID="bcd48699-8090-4574-934c-b29668788243" Nov 02 01:45:16.985925 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:16.985906 4427 scope.go:122] "RemoveContainer" containerID="268b5b0735e3ff50481c3aa866c42c2aad40e4ea8b8516dbe442c52cdeba3e7b" Nov 02 01:45:16.990928 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:16.990907 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893065-ltmvv"] Nov 02 01:45:17.005568 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:17.005536 4427 scope.go:122] "RemoveContainer" containerID="2d569355d0d8a976f794fb3315cde993d20d52282d73b571220f475284f311ab" Nov 02 01:45:17.235158 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:17.235074 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893065-ltmvv" Nov 02 01:45:17.311705 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:17.311668 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/635a1467-2a08-4b8b-a8c0-b16e2a361888-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/635a1467-2a08-4b8b-a8c0-b16e2a361888-maas-api-service-ca\") pod \"635a1467-2a08-4b8b-a8c0-b16e2a361888\" (UID: \"635a1467-2a08-4b8b-a8c0-b16e2a361888\") " Nov 02 01:45:17.311912 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:17.311876 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/635a1467-2a08-4b8b-a8c0-b16e2a361888-kube-api-access-rm9nd\" (UniqueName: \"kubernetes.io/projected/635a1467-2a08-4b8b-a8c0-b16e2a361888-kube-api-access-rm9nd\") pod \"635a1467-2a08-4b8b-a8c0-b16e2a361888\" (UID: \"635a1467-2a08-4b8b-a8c0-b16e2a361888\") " Nov 02 01:45:17.312088 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:17.312065 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/635a1467-2a08-4b8b-a8c0-b16e2a361888-maas-api-service-ca" pod "635a1467-2a08-4b8b-a8c0-b16e2a361888" (UID: "635a1467-2a08-4b8b-a8c0-b16e2a361888"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:45:17.312598 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:17.312569 4427 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/635a1467-2a08-4b8b-a8c0-b16e2a361888-maas-api-service-ca\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:17.314584 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:17.314552 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/635a1467-2a08-4b8b-a8c0-b16e2a361888-kube-api-access-rm9nd" pod "635a1467-2a08-4b8b-a8c0-b16e2a361888" (UID: "635a1467-2a08-4b8b-a8c0-b16e2a361888"). InnerVolumeSpecName "kube-api-access-rm9nd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:45:17.414563 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:17.414528 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm9nd\" (UniqueName: \"kubernetes.io/projected/635a1467-2a08-4b8b-a8c0-b16e2a361888-kube-api-access-rm9nd\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:17.544024 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:17.541601 4427 scope.go:122] "RemoveContainer" containerID="2ddce910e8a040b7ba324270b94b58b22551871a1a251846d05b368b2cb445b2" Nov 02 01:45:17.544443 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:17.544419 4427 scope.go:122] "RemoveContainer" containerID="e6ebbd7bf01c918acd10d627f57b0dc157e54597ccf49541619167b0ebebe413" Nov 02 01:45:17.989972 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:17.989937 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893065-ltmvv" event={"ID":"635a1467-2a08-4b8b-a8c0-b16e2a361888","Type":"ContainerDied","Data":"e5d68a3b98a0eddeba23af77cfbdff705206f25d56573471377c1a933e2c27c1"} Nov 02 01:45:17.990495 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:17.990002 4427 scope.go:122] "RemoveContainer" containerID="4efbe14aca4d71912deb75c5851640b908c912b3f1293b99de2c1316ff19ba26" Nov 02 01:45:17.990495 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:17.989950 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893065-ltmvv" Nov 02 01:45:17.994305 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:17.994277 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0f32d6-29893065-5kg4t" event={"ID":"10bfbdc2-73c3-43b5-9087-bc07be95319e","Type":"ContainerStarted","Data":"81a4ef642a2935d89ecae14ac36e9c2270a3c5bf35e3233bf381d2bbe2bb5601"} Nov 02 01:45:17.996598 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:17.996575 4427 generic.go:356] "Generic (PLEG): container finished" podID="d983a998-461c-4341-aea9-1eb65375bbb7" containerID="32dd69a55f6f492b14a7b2f124f09a1997ddfb85f09058710a37707713795521" exitCode=60 Nov 02 01:45:17.996701 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:17.996646 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e06f2-29893065-4ql7w" event={"ID":"d983a998-461c-4341-aea9-1eb65375bbb7","Type":"ContainerDied","Data":"32dd69a55f6f492b14a7b2f124f09a1997ddfb85f09058710a37707713795521"} Nov 02 01:45:17.997031 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:17.997017 4427 scope.go:122] "RemoveContainer" containerID="32dd69a55f6f492b14a7b2f124f09a1997ddfb85f09058710a37707713795521" Nov 02 01:45:17.997337 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:45:17.997319 4427 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-0e06f2-29893065-4ql7w_odh-ai-gateway-infra(d983a998-461c-4341-aea9-1eb65375bbb7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e06f2-29893065-4ql7w" podUID="d983a998-461c-4341-aea9-1eb65375bbb7" Nov 02 01:45:18.001716 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:18.001695 4427 scope.go:122] "RemoveContainer" containerID="65af2666321da9f96376b9d2addfdd750a831c03963345371127aa05f749ef91" Nov 02 01:45:18.006957 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:18.006929 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-123986-29893065-dvhsz"] Nov 02 01:45:18.018528 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:18.018492 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893065-ltmvv"] Nov 02 01:45:18.027475 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:18.027441 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ca1c53-29893065-ltmvv"] Nov 02 01:45:18.031238 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:18.031208 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-00f1da-29893065-vkvdj"] Nov 02 01:45:18.045027 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:18.044977 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9d816-29893065-vvz6j"] Nov 02 01:45:18.251721 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:18.251687 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-123986-29893065-dvhsz" Nov 02 01:45:18.297776 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:18.297747 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-00f1da-29893065-vkvdj" Nov 02 01:45:18.312335 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:18.312309 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9d816-29893065-vvz6j" Nov 02 01:45:18.359282 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:18.359211 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7ebe1a8-c572-4b18-98d9-083b497aa758-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7ebe1a8-c572-4b18-98d9-083b497aa758-maas-api-service-ca\") pod \"c7ebe1a8-c572-4b18-98d9-083b497aa758\" (UID: \"c7ebe1a8-c572-4b18-98d9-083b497aa758\") " Nov 02 01:45:18.359576 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:18.359555 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7ebe1a8-c572-4b18-98d9-083b497aa758-kube-api-access-wgqk7\" (UniqueName: \"kubernetes.io/projected/c7ebe1a8-c572-4b18-98d9-083b497aa758-kube-api-access-wgqk7\") pod \"c7ebe1a8-c572-4b18-98d9-083b497aa758\" (UID: \"c7ebe1a8-c572-4b18-98d9-083b497aa758\") " Nov 02 01:45:18.361281 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:18.361248 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7ebe1a8-c572-4b18-98d9-083b497aa758-maas-api-service-ca" pod "c7ebe1a8-c572-4b18-98d9-083b497aa758" (UID: "c7ebe1a8-c572-4b18-98d9-083b497aa758"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:45:18.367944 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:18.367885 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7ebe1a8-c572-4b18-98d9-083b497aa758-kube-api-access-wgqk7" pod "c7ebe1a8-c572-4b18-98d9-083b497aa758" (UID: "c7ebe1a8-c572-4b18-98d9-083b497aa758"). InnerVolumeSpecName "kube-api-access-wgqk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:45:18.461141 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:18.461045 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5779a226-eaa5-4428-99d1-47e4936591a2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5779a226-eaa5-4428-99d1-47e4936591a2-maas-api-service-ca\") pod \"5779a226-eaa5-4428-99d1-47e4936591a2\" (UID: \"5779a226-eaa5-4428-99d1-47e4936591a2\") " Nov 02 01:45:18.461141 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:18.461110 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5779a226-eaa5-4428-99d1-47e4936591a2-kube-api-access-p5gdd\" (UniqueName: \"kubernetes.io/projected/5779a226-eaa5-4428-99d1-47e4936591a2-kube-api-access-p5gdd\") pod \"5779a226-eaa5-4428-99d1-47e4936591a2\" (UID: \"5779a226-eaa5-4428-99d1-47e4936591a2\") " Nov 02 01:45:18.461438 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:18.461418 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcd48699-8090-4574-934c-b29668788243-kube-api-access-cvm8x\" (UniqueName: \"kubernetes.io/projected/bcd48699-8090-4574-934c-b29668788243-kube-api-access-cvm8x\") pod \"bcd48699-8090-4574-934c-b29668788243\" (UID: \"bcd48699-8090-4574-934c-b29668788243\") " Nov 02 01:45:18.461506 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:18.461456 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcd48699-8090-4574-934c-b29668788243-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcd48699-8090-4574-934c-b29668788243-maas-api-service-ca\") pod \"bcd48699-8090-4574-934c-b29668788243\" (UID: \"bcd48699-8090-4574-934c-b29668788243\") " Nov 02 01:45:18.461713 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:18.461668 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5779a226-eaa5-4428-99d1-47e4936591a2-maas-api-service-ca" pod "5779a226-eaa5-4428-99d1-47e4936591a2" (UID: "5779a226-eaa5-4428-99d1-47e4936591a2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:45:18.462053 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:18.462024 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcd48699-8090-4574-934c-b29668788243-maas-api-service-ca" pod "bcd48699-8090-4574-934c-b29668788243" (UID: "bcd48699-8090-4574-934c-b29668788243"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:45:18.462338 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:18.462314 4427 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcd48699-8090-4574-934c-b29668788243-maas-api-service-ca\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:18.462453 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:18.462345 4427 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7ebe1a8-c572-4b18-98d9-083b497aa758-maas-api-service-ca\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:18.462453 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:18.462364 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgqk7\" (UniqueName: \"kubernetes.io/projected/c7ebe1a8-c572-4b18-98d9-083b497aa758-kube-api-access-wgqk7\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:18.462453 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:18.462404 4427 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5779a226-eaa5-4428-99d1-47e4936591a2-maas-api-service-ca\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:18.463608 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:18.463585 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5779a226-eaa5-4428-99d1-47e4936591a2-kube-api-access-p5gdd" pod "5779a226-eaa5-4428-99d1-47e4936591a2" (UID: "5779a226-eaa5-4428-99d1-47e4936591a2"). InnerVolumeSpecName "kube-api-access-p5gdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:45:18.463820 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:18.463803 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcd48699-8090-4574-934c-b29668788243-kube-api-access-cvm8x" pod "bcd48699-8090-4574-934c-b29668788243" (UID: "bcd48699-8090-4574-934c-b29668788243"). InnerVolumeSpecName "kube-api-access-cvm8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:45:18.527650 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:18.527619 4427 scope.go:122] "RemoveContainer" containerID="4acc183591b34d16dfbaa53855f28c500c0acbfb227c3104e5a2a57f117ad44e" Nov 02 01:45:18.564815 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:18.564779 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvm8x\" (UniqueName: \"kubernetes.io/projected/bcd48699-8090-4574-934c-b29668788243-kube-api-access-cvm8x\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:18.564815 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:18.564817 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5gdd\" (UniqueName: \"kubernetes.io/projected/5779a226-eaa5-4428-99d1-47e4936591a2-kube-api-access-p5gdd\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:19.006403 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:19.006363 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-123986-29893065-dvhsz" Nov 02 01:45:19.006820 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:19.006362 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-123986-29893065-dvhsz" event={"ID":"c7ebe1a8-c572-4b18-98d9-083b497aa758","Type":"ContainerDied","Data":"2ed4b4053ab25d641b88e0a75d971581e52045f7d7bcee7fcaaa6d9b6388e560"} Nov 02 01:45:19.006820 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:19.006504 4427 scope.go:122] "RemoveContainer" containerID="a1e9e78ca81b59ebdd95b5980f3be099da3590aee7f0d02449196288e6b2cb49" Nov 02 01:45:19.008166 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:19.008136 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-00f1da-29893065-vkvdj" event={"ID":"5779a226-eaa5-4428-99d1-47e4936591a2","Type":"ContainerDied","Data":"9cffdc73841c9ac4b4022998f38c8108cf16e54b0fc9ab892c9142971d51098f"} Nov 02 01:45:19.008314 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:19.008231 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-00f1da-29893065-vkvdj" Nov 02 01:45:19.011362 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:19.011300 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9d816-29893065-vvz6j" event={"ID":"bcd48699-8090-4574-934c-b29668788243","Type":"ContainerDied","Data":"a5e907619671644f7c1ac90274b883e2bcda30974d95593f1c308983b4388085"} Nov 02 01:45:19.011362 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:19.011315 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9d816-29893065-vvz6j" Nov 02 01:45:19.018212 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:19.018050 4427 generic.go:356] "Generic (PLEG): container finished" podID="886fa3e6-9143-4823-903d-7e68d06100ee" containerID="8aa6011766120e972d4f45477875e3b89a1fa36672efa93503129613f498cd0d" exitCode=60 Nov 02 01:45:19.018743 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:19.018718 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3c4ea5-29893065-d2hkv" event={"ID":"886fa3e6-9143-4823-903d-7e68d06100ee","Type":"ContainerDied","Data":"8aa6011766120e972d4f45477875e3b89a1fa36672efa93503129613f498cd0d"} Nov 02 01:45:19.021145 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:19.021017 4427 scope.go:122] "RemoveContainer" containerID="9bb38d2157dddb649b4cacb4610ec2845df8795070d438a619635c16f4f670c3" Nov 02 01:45:19.021145 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:19.021087 4427 scope.go:122] "RemoveContainer" containerID="8aa6011766120e972d4f45477875e3b89a1fa36672efa93503129613f498cd0d" Nov 02 01:45:19.021863 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:45:19.021835 4427 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-3c4ea5-29893065-d2hkv_odh-ai-gateway-infra(886fa3e6-9143-4823-903d-7e68d06100ee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3c4ea5-29893065-d2hkv" podUID="886fa3e6-9143-4823-903d-7e68d06100ee" Nov 02 01:45:19.034974 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:19.034938 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bbb385-29893065-wl5kl" event={"ID":"747eccfe-eecb-4cec-99a9-aea7857e1a34","Type":"ContainerStarted","Data":"0cade1a6e5fefd363d2dd8f6bc51c8af3b04a08a1df2ff7a5f2aa649492ca4f7"} Nov 02 01:45:19.041244 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:19.041218 4427 generic.go:356] "Generic (PLEG): container finished" podID="10bfbdc2-73c3-43b5-9087-bc07be95319e" containerID="81a4ef642a2935d89ecae14ac36e9c2270a3c5bf35e3233bf381d2bbe2bb5601" exitCode=60 Nov 02 01:45:19.044434 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:19.041307 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0f32d6-29893065-5kg4t" event={"ID":"10bfbdc2-73c3-43b5-9087-bc07be95319e","Type":"ContainerDied","Data":"81a4ef642a2935d89ecae14ac36e9c2270a3c5bf35e3233bf381d2bbe2bb5601"} Nov 02 01:45:19.044434 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:19.042728 4427 scope.go:122] "RemoveContainer" containerID="81a4ef642a2935d89ecae14ac36e9c2270a3c5bf35e3233bf381d2bbe2bb5601" Nov 02 01:45:19.044434 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:45:19.043075 4427 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-0f32d6-29893065-5kg4t_odh-ai-gateway-infra(10bfbdc2-73c3-43b5-9087-bc07be95319e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0f32d6-29893065-5kg4t" podUID="10bfbdc2-73c3-43b5-9087-bc07be95319e" Nov 02 01:45:19.048445 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:19.048426 4427 scope.go:122] "RemoveContainer" containerID="2d20a65009436527a20d88f10ac9a1cb6f51de8c535e9c9680021020e5cf480c" Nov 02 01:45:19.050945 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:19.050919 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-00f1da-29893065-vkvdj"] Nov 02 01:45:19.054745 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:19.054721 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-00f1da-29893065-vkvdj"] Nov 02 01:45:19.059141 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:19.059121 4427 scope.go:122] "RemoveContainer" containerID="e6ebbd7bf01c918acd10d627f57b0dc157e54597ccf49541619167b0ebebe413" Nov 02 01:45:19.065977 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:19.065955 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0f32d6-29893065-5kg4t"] Nov 02 01:45:19.074950 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:19.074926 4427 scope.go:122] "RemoveContainer" containerID="2ddce910e8a040b7ba324270b94b58b22551871a1a251846d05b368b2cb445b2" Nov 02 01:45:19.087861 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:19.087831 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e06f2-29893065-4ql7w"] Nov 02 01:45:19.100845 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:19.100817 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-123986-29893065-dvhsz"] Nov 02 01:45:19.106965 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:19.106243 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-123986-29893065-dvhsz"] Nov 02 01:45:19.118434 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:19.118403 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9d816-29893065-vvz6j"] Nov 02 01:45:19.124883 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:19.124857 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b9d816-29893065-vvz6j"] Nov 02 01:45:19.320754 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:19.320726 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e06f2-29893065-4ql7w" Nov 02 01:45:19.479830 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:19.479797 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d983a998-461c-4341-aea9-1eb65375bbb7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d983a998-461c-4341-aea9-1eb65375bbb7-maas-api-service-ca\") pod \"d983a998-461c-4341-aea9-1eb65375bbb7\" (UID: \"d983a998-461c-4341-aea9-1eb65375bbb7\") " Nov 02 01:45:19.479976 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:19.479957 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d983a998-461c-4341-aea9-1eb65375bbb7-kube-api-access-hkt5q\" (UniqueName: \"kubernetes.io/projected/d983a998-461c-4341-aea9-1eb65375bbb7-kube-api-access-hkt5q\") pod \"d983a998-461c-4341-aea9-1eb65375bbb7\" (UID: \"d983a998-461c-4341-aea9-1eb65375bbb7\") " Nov 02 01:45:19.480234 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:19.480209 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d983a998-461c-4341-aea9-1eb65375bbb7-maas-api-service-ca" pod "d983a998-461c-4341-aea9-1eb65375bbb7" (UID: "d983a998-461c-4341-aea9-1eb65375bbb7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:45:19.480552 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:19.480530 4427 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d983a998-461c-4341-aea9-1eb65375bbb7-maas-api-service-ca\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:19.482171 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:19.482151 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d983a998-461c-4341-aea9-1eb65375bbb7-kube-api-access-hkt5q" pod "d983a998-461c-4341-aea9-1eb65375bbb7" (UID: "d983a998-461c-4341-aea9-1eb65375bbb7"). InnerVolumeSpecName "kube-api-access-hkt5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:45:19.537014 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:19.536928 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5779a226-eaa5-4428-99d1-47e4936591a2" path="/var/lib/kubelet/pods/5779a226-eaa5-4428-99d1-47e4936591a2/volumes" Nov 02 01:45:19.537467 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:19.537452 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="635a1467-2a08-4b8b-a8c0-b16e2a361888" path="/var/lib/kubelet/pods/635a1467-2a08-4b8b-a8c0-b16e2a361888/volumes" Nov 02 01:45:19.537949 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:19.537936 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcd48699-8090-4574-934c-b29668788243" path="/var/lib/kubelet/pods/bcd48699-8090-4574-934c-b29668788243/volumes" Nov 02 01:45:19.538648 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:19.538633 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7ebe1a8-c572-4b18-98d9-083b497aa758" path="/var/lib/kubelet/pods/c7ebe1a8-c572-4b18-98d9-083b497aa758/volumes" Nov 02 01:45:19.582130 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:19.582094 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkt5q\" (UniqueName: \"kubernetes.io/projected/d983a998-461c-4341-aea9-1eb65375bbb7-kube-api-access-hkt5q\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:20.091315 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:20.091278 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3c4ea5-29893065-d2hkv"] Nov 02 01:45:20.157426 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:20.155282 4427 generic.go:356] "Generic (PLEG): container finished" podID="747eccfe-eecb-4cec-99a9-aea7857e1a34" containerID="0cade1a6e5fefd363d2dd8f6bc51c8af3b04a08a1df2ff7a5f2aa649492ca4f7" exitCode=60 Nov 02 01:45:20.157426 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:20.155448 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bbb385-29893065-wl5kl" event={"ID":"747eccfe-eecb-4cec-99a9-aea7857e1a34","Type":"ContainerDied","Data":"0cade1a6e5fefd363d2dd8f6bc51c8af3b04a08a1df2ff7a5f2aa649492ca4f7"} Nov 02 01:45:20.157426 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:20.155509 4427 scope.go:122] "RemoveContainer" containerID="4acc183591b34d16dfbaa53855f28c500c0acbfb227c3104e5a2a57f117ad44e" Nov 02 01:45:20.157426 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:20.155998 4427 scope.go:122] "RemoveContainer" containerID="0cade1a6e5fefd363d2dd8f6bc51c8af3b04a08a1df2ff7a5f2aa649492ca4f7" Nov 02 01:45:20.157426 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:45:20.156445 4427 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-bbb385-29893065-wl5kl_odh-ai-gateway-infra(747eccfe-eecb-4cec-99a9-aea7857e1a34)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bbb385-29893065-wl5kl" podUID="747eccfe-eecb-4cec-99a9-aea7857e1a34" Nov 02 01:45:20.177703 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:20.177668 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e06f2-29893065-4ql7w" event={"ID":"d983a998-461c-4341-aea9-1eb65375bbb7","Type":"ContainerDied","Data":"59df0cffd315d944f8acf17d9881406903fa197cb5b8a9292213108ddc14d6b3"} Nov 02 01:45:20.177997 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:20.177980 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e06f2-29893065-4ql7w" Nov 02 01:45:20.233497 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:20.232824 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bbb385-29893065-wl5kl"] Nov 02 01:45:20.265863 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:20.265838 4427 scope.go:122] "RemoveContainer" containerID="32dd69a55f6f492b14a7b2f124f09a1997ddfb85f09058710a37707713795521" Nov 02 01:45:20.296544 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:20.294297 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-12e7cb/test-model-12e7cb-kserve-5b5d7468c9-xkkqs"] Nov 02 01:45:20.385118 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:20.385051 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e06f2-29893065-4ql7w"] Nov 02 01:45:20.398423 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:20.398212 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e06f2-29893065-4ql7w"] Nov 02 01:45:20.754471 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:20.754444 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3c4ea5-29893065-d2hkv" Nov 02 01:45:20.895607 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:20.895530 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0f32d6-29893065-5kg4t" Nov 02 01:45:20.899446 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:20.899416 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/886fa3e6-9143-4823-903d-7e68d06100ee-kube-api-access-j4xhd\" (UniqueName: \"kubernetes.io/projected/886fa3e6-9143-4823-903d-7e68d06100ee-kube-api-access-j4xhd\") pod \"886fa3e6-9143-4823-903d-7e68d06100ee\" (UID: \"886fa3e6-9143-4823-903d-7e68d06100ee\") " Nov 02 01:45:20.899837 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:20.899813 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/886fa3e6-9143-4823-903d-7e68d06100ee-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/886fa3e6-9143-4823-903d-7e68d06100ee-maas-api-service-ca\") pod \"886fa3e6-9143-4823-903d-7e68d06100ee\" (UID: \"886fa3e6-9143-4823-903d-7e68d06100ee\") " Nov 02 01:45:20.900497 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:20.900467 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/886fa3e6-9143-4823-903d-7e68d06100ee-maas-api-service-ca" pod "886fa3e6-9143-4823-903d-7e68d06100ee" (UID: "886fa3e6-9143-4823-903d-7e68d06100ee"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:45:20.901756 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:20.901734 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/886fa3e6-9143-4823-903d-7e68d06100ee-kube-api-access-j4xhd" pod "886fa3e6-9143-4823-903d-7e68d06100ee" (UID: "886fa3e6-9143-4823-903d-7e68d06100ee"). InnerVolumeSpecName "kube-api-access-j4xhd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:45:21.001409 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:21.001355 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10bfbdc2-73c3-43b5-9087-bc07be95319e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10bfbdc2-73c3-43b5-9087-bc07be95319e-maas-api-service-ca\") pod \"10bfbdc2-73c3-43b5-9087-bc07be95319e\" (UID: \"10bfbdc2-73c3-43b5-9087-bc07be95319e\") " Nov 02 01:45:21.001602 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:21.001583 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10bfbdc2-73c3-43b5-9087-bc07be95319e-kube-api-access-j26x6\" (UniqueName: \"kubernetes.io/projected/10bfbdc2-73c3-43b5-9087-bc07be95319e-kube-api-access-j26x6\") pod \"10bfbdc2-73c3-43b5-9087-bc07be95319e\" (UID: \"10bfbdc2-73c3-43b5-9087-bc07be95319e\") " Nov 02 01:45:21.001788 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:21.001763 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10bfbdc2-73c3-43b5-9087-bc07be95319e-maas-api-service-ca" pod "10bfbdc2-73c3-43b5-9087-bc07be95319e" (UID: "10bfbdc2-73c3-43b5-9087-bc07be95319e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:45:21.002170 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:21.002147 4427 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/886fa3e6-9143-4823-903d-7e68d06100ee-maas-api-service-ca\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:21.002247 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:21.002173 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4xhd\" (UniqueName: \"kubernetes.io/projected/886fa3e6-9143-4823-903d-7e68d06100ee-kube-api-access-j4xhd\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:21.002247 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:21.002182 4427 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10bfbdc2-73c3-43b5-9087-bc07be95319e-maas-api-service-ca\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:21.005114 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:21.005087 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10bfbdc2-73c3-43b5-9087-bc07be95319e-kube-api-access-j26x6" pod "10bfbdc2-73c3-43b5-9087-bc07be95319e" (UID: "10bfbdc2-73c3-43b5-9087-bc07be95319e"). InnerVolumeSpecName "kube-api-access-j26x6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:45:21.103551 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:21.103510 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j26x6\" (UniqueName: \"kubernetes.io/projected/10bfbdc2-73c3-43b5-9087-bc07be95319e-kube-api-access-j26x6\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:21.222669 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:21.222582 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3c4ea5-29893065-d2hkv" Nov 02 01:45:21.222822 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:21.222585 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3c4ea5-29893065-d2hkv" event={"ID":"886fa3e6-9143-4823-903d-7e68d06100ee","Type":"ContainerDied","Data":"7f5528f3813d7e603ddabf9af36f6de066308994f86dfbc3e28d5a042eb442a2"} Nov 02 01:45:21.222899 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:21.222853 4427 scope.go:122] "RemoveContainer" containerID="8aa6011766120e972d4f45477875e3b89a1fa36672efa93503129613f498cd0d" Nov 02 01:45:21.226274 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:21.226228 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0f32d6-29893065-5kg4t" event={"ID":"10bfbdc2-73c3-43b5-9087-bc07be95319e","Type":"ContainerDied","Data":"9b49c1fa05c0f968c077b1147ad2588061b21a323390a56201c6078ef7a9670f"} Nov 02 01:45:21.226426 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:21.226262 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0f32d6-29893065-5kg4t" Nov 02 01:45:21.241691 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:21.240785 4427 scope.go:122] "RemoveContainer" containerID="81a4ef642a2935d89ecae14ac36e9c2270a3c5bf35e3233bf381d2bbe2bb5601" Nov 02 01:45:21.272422 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:21.271038 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3c4ea5-29893065-d2hkv"] Nov 02 01:45:21.276529 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:21.276504 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3c4ea5-29893065-d2hkv"] Nov 02 01:45:21.290097 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:21.290070 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0f32d6-29893065-5kg4t"] Nov 02 01:45:21.300125 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:21.300097 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0f32d6-29893065-5kg4t"] Nov 02 01:45:21.387020 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:21.386985 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bbb385-29893065-wl5kl" Nov 02 01:45:21.511354 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:21.511257 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/747eccfe-eecb-4cec-99a9-aea7857e1a34-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/747eccfe-eecb-4cec-99a9-aea7857e1a34-maas-api-service-ca\") pod \"747eccfe-eecb-4cec-99a9-aea7857e1a34\" (UID: \"747eccfe-eecb-4cec-99a9-aea7857e1a34\") " Nov 02 01:45:21.511354 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:21.511321 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/747eccfe-eecb-4cec-99a9-aea7857e1a34-kube-api-access-w48j9\" (UniqueName: \"kubernetes.io/projected/747eccfe-eecb-4cec-99a9-aea7857e1a34-kube-api-access-w48j9\") pod \"747eccfe-eecb-4cec-99a9-aea7857e1a34\" (UID: \"747eccfe-eecb-4cec-99a9-aea7857e1a34\") " Nov 02 01:45:21.511691 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:21.511662 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/747eccfe-eecb-4cec-99a9-aea7857e1a34-maas-api-service-ca" pod "747eccfe-eecb-4cec-99a9-aea7857e1a34" (UID: "747eccfe-eecb-4cec-99a9-aea7857e1a34"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:45:21.512005 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:21.511970 4427 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/747eccfe-eecb-4cec-99a9-aea7857e1a34-maas-api-service-ca\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:21.513888 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:21.513861 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/747eccfe-eecb-4cec-99a9-aea7857e1a34-kube-api-access-w48j9" pod "747eccfe-eecb-4cec-99a9-aea7857e1a34" (UID: "747eccfe-eecb-4cec-99a9-aea7857e1a34"). InnerVolumeSpecName "kube-api-access-w48j9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:45:21.535535 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:21.535506 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10bfbdc2-73c3-43b5-9087-bc07be95319e" path="/var/lib/kubelet/pods/10bfbdc2-73c3-43b5-9087-bc07be95319e/volumes" Nov 02 01:45:21.535982 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:21.535969 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="886fa3e6-9143-4823-903d-7e68d06100ee" path="/var/lib/kubelet/pods/886fa3e6-9143-4823-903d-7e68d06100ee/volumes" Nov 02 01:45:21.536351 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:21.536339 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d983a998-461c-4341-aea9-1eb65375bbb7" path="/var/lib/kubelet/pods/d983a998-461c-4341-aea9-1eb65375bbb7/volumes" Nov 02 01:45:21.616965 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:21.616925 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w48j9\" (UniqueName: \"kubernetes.io/projected/747eccfe-eecb-4cec-99a9-aea7857e1a34-kube-api-access-w48j9\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:22.276399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:22.276286 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bbb385-29893065-wl5kl" event={"ID":"747eccfe-eecb-4cec-99a9-aea7857e1a34","Type":"ContainerDied","Data":"af6b5a0758b80dffc44f7cc4f33c4d925cd28c49a44de509a37351d8fc0347b6"} Nov 02 01:45:22.276399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:22.276360 4427 scope.go:122] "RemoveContainer" containerID="0cade1a6e5fefd363d2dd8f6bc51c8af3b04a08a1df2ff7a5f2aa649492ca4f7" Nov 02 01:45:22.276917 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:22.276476 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bbb385-29893065-wl5kl" Nov 02 01:45:22.357529 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:22.357444 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bbb385-29893065-wl5kl"] Nov 02 01:45:22.368410 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:22.367899 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bbb385-29893065-wl5kl"] Nov 02 01:45:23.536619 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:23.536587 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="747eccfe-eecb-4cec-99a9-aea7857e1a34" path="/var/lib/kubelet/pods/747eccfe-eecb-4cec-99a9-aea7857e1a34/volumes" Nov 02 01:45:28.188988 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.183549 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd"] Nov 02 01:45:28.188988 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.183785 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd" podUID="d9f64fd6-8cb1-4a34-99f0-828c753a1119" containerName="istio-proxy" containerID="cri-o://ba3e2481997f8ddb323e8b6862e4cf4a4bd794e8f35a8e774e0aae11265a8a5a" gracePeriod=30 Nov 02 01:45:28.198223 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.192502 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm"] Nov 02 01:45:28.198223 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.192735 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm" podUID="136eabea-7a63-4d89-bcc1-0e3d6c8810cf" containerName="istio-proxy" containerID="cri-o://e7bad34c068724ee09f318ec5119e242811f511e6d11437373e7b874c693f60a" gracePeriod=30 Nov 02 01:45:28.198223 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.197472 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt"] Nov 02 01:45:28.198223 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.197704 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt" podUID="82d82a81-f1df-446d-b07b-20f510354a7e" containerName="istio-proxy" containerID="cri-o://1f7f09782a19394d9cf33bb0bd9e856f43155b286d13912c18e7d0b10098df76" gracePeriod=30 Nov 02 01:45:28.207964 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.207933 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj"] Nov 02 01:45:28.208346 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.208320 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj" podUID="36cac416-8352-4653-8ec9-7ad64348afa7" containerName="istio-proxy" containerID="cri-o://cf5643a4140429f936a509f8ea84de25169cef9edca00cf73746c2659bd2a183" gracePeriod=30 Nov 02 01:45:28.393397 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.391313 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht"] Nov 02 01:45:28.432263 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.432193 4427 cpu_manager.go:414] "containerMap: removing container" podUID="635a1467-2a08-4b8b-a8c0-b16e2a361888" containerName="cleanup" Nov 02 01:45:28.432263 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.432225 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="635a1467-2a08-4b8b-a8c0-b16e2a361888" containerName="cleanup" Nov 02 01:45:28.432263 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.432240 4427 cpu_manager.go:414] "containerMap: removing container" podUID="747eccfe-eecb-4cec-99a9-aea7857e1a34" containerName="cleanup" Nov 02 01:45:28.432263 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.432248 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="747eccfe-eecb-4cec-99a9-aea7857e1a34" containerName="cleanup" Nov 02 01:45:28.432263 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.432260 4427 cpu_manager.go:414] "containerMap: removing container" podUID="5779a226-eaa5-4428-99d1-47e4936591a2" containerName="cleanup" Nov 02 01:45:28.432263 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.432268 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5779a226-eaa5-4428-99d1-47e4936591a2" containerName="cleanup" Nov 02 01:45:28.432754 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.432286 4427 cpu_manager.go:414] "containerMap: removing container" podUID="d983a998-461c-4341-aea9-1eb65375bbb7" containerName="cleanup" Nov 02 01:45:28.432754 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.432293 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d983a998-461c-4341-aea9-1eb65375bbb7" containerName="cleanup" Nov 02 01:45:28.432754 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.432306 4427 cpu_manager.go:414] "containerMap: removing container" podUID="bcd48699-8090-4574-934c-b29668788243" containerName="cleanup" Nov 02 01:45:28.432754 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.432313 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcd48699-8090-4574-934c-b29668788243" containerName="cleanup" Nov 02 01:45:28.432754 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.432323 4427 cpu_manager.go:414] "containerMap: removing container" podUID="886fa3e6-9143-4823-903d-7e68d06100ee" containerName="cleanup" Nov 02 01:45:28.432754 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.432331 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="886fa3e6-9143-4823-903d-7e68d06100ee" containerName="cleanup" Nov 02 01:45:28.432754 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.432344 4427 cpu_manager.go:414] "containerMap: removing container" podUID="af735a61-695b-4a66-a7cb-d0df06b3b58b" containerName="istio-proxy" Nov 02 01:45:28.432754 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.432351 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af735a61-695b-4a66-a7cb-d0df06b3b58b" containerName="istio-proxy" Nov 02 01:45:28.432754 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.432368 4427 cpu_manager.go:414] "containerMap: removing container" podUID="5779a226-eaa5-4428-99d1-47e4936591a2" containerName="cleanup" Nov 02 01:45:28.432754 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.432395 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5779a226-eaa5-4428-99d1-47e4936591a2" containerName="cleanup" Nov 02 01:45:28.432754 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.432404 4427 cpu_manager.go:414] "containerMap: removing container" podUID="d983a998-461c-4341-aea9-1eb65375bbb7" containerName="cleanup" Nov 02 01:45:28.432754 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.432411 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d983a998-461c-4341-aea9-1eb65375bbb7" containerName="cleanup" Nov 02 01:45:28.432754 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.432421 4427 cpu_manager.go:414] "containerMap: removing container" podUID="635a1467-2a08-4b8b-a8c0-b16e2a361888" containerName="cleanup" Nov 02 01:45:28.432754 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.432428 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="635a1467-2a08-4b8b-a8c0-b16e2a361888" containerName="cleanup" Nov 02 01:45:28.432754 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.432436 4427 cpu_manager.go:414] "containerMap: removing container" podUID="bcd48699-8090-4574-934c-b29668788243" containerName="cleanup" Nov 02 01:45:28.432754 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.432443 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcd48699-8090-4574-934c-b29668788243" containerName="cleanup" Nov 02 01:45:28.432754 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.432467 4427 cpu_manager.go:414] "containerMap: removing container" podUID="d983a998-461c-4341-aea9-1eb65375bbb7" containerName="cleanup" Nov 02 01:45:28.432754 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.432474 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d983a998-461c-4341-aea9-1eb65375bbb7" containerName="cleanup" Nov 02 01:45:28.432754 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.432493 4427 cpu_manager.go:414] "containerMap: removing container" podUID="886fa3e6-9143-4823-903d-7e68d06100ee" containerName="cleanup" Nov 02 01:45:28.432754 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.432502 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="886fa3e6-9143-4823-903d-7e68d06100ee" containerName="cleanup" Nov 02 01:45:28.432754 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.432522 4427 cpu_manager.go:414] "containerMap: removing container" podUID="c7ebe1a8-c572-4b18-98d9-083b497aa758" containerName="cleanup" Nov 02 01:45:28.432754 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.432528 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7ebe1a8-c572-4b18-98d9-083b497aa758" containerName="cleanup" Nov 02 01:45:28.432754 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.432544 4427 cpu_manager.go:414] "containerMap: removing container" podUID="635a1467-2a08-4b8b-a8c0-b16e2a361888" containerName="cleanup" Nov 02 01:45:28.432754 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.432551 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="635a1467-2a08-4b8b-a8c0-b16e2a361888" containerName="cleanup" Nov 02 01:45:28.432754 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.432568 4427 cpu_manager.go:414] "containerMap: removing container" podUID="c7ebe1a8-c572-4b18-98d9-083b497aa758" containerName="cleanup" Nov 02 01:45:28.432754 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.432575 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7ebe1a8-c572-4b18-98d9-083b497aa758" containerName="cleanup" Nov 02 01:45:28.432754 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.432591 4427 cpu_manager.go:414] "containerMap: removing container" podUID="c7ebe1a8-c572-4b18-98d9-083b497aa758" containerName="cleanup" Nov 02 01:45:28.432754 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.432599 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7ebe1a8-c572-4b18-98d9-083b497aa758" containerName="cleanup" Nov 02 01:45:28.432754 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.432608 4427 cpu_manager.go:414] "containerMap: removing container" podUID="747eccfe-eecb-4cec-99a9-aea7857e1a34" containerName="cleanup" Nov 02 01:45:28.432754 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.432615 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="747eccfe-eecb-4cec-99a9-aea7857e1a34" containerName="cleanup" Nov 02 01:45:28.432754 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.432626 4427 cpu_manager.go:414] "containerMap: removing container" podUID="747eccfe-eecb-4cec-99a9-aea7857e1a34" containerName="cleanup" Nov 02 01:45:28.432754 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.432633 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="747eccfe-eecb-4cec-99a9-aea7857e1a34" containerName="cleanup" Nov 02 01:45:28.432754 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.432647 4427 cpu_manager.go:414] "containerMap: removing container" podUID="10bfbdc2-73c3-43b5-9087-bc07be95319e" containerName="cleanup" Nov 02 01:45:28.432754 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.432654 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10bfbdc2-73c3-43b5-9087-bc07be95319e" containerName="cleanup" Nov 02 01:45:28.432754 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.432671 4427 cpu_manager.go:414] "containerMap: removing container" podUID="f3730ee2-6ba0-4dec-b103-214881818132" containerName="istio-proxy" Nov 02 01:45:28.432754 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.432677 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3730ee2-6ba0-4dec-b103-214881818132" containerName="istio-proxy" Nov 02 01:45:28.432754 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.432688 4427 cpu_manager.go:414] "containerMap: removing container" podUID="10bfbdc2-73c3-43b5-9087-bc07be95319e" containerName="cleanup" Nov 02 01:45:28.432754 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.432695 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10bfbdc2-73c3-43b5-9087-bc07be95319e" containerName="cleanup" Nov 02 01:45:28.432754 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.432705 4427 cpu_manager.go:414] "containerMap: removing container" podUID="10bfbdc2-73c3-43b5-9087-bc07be95319e" containerName="cleanup" Nov 02 01:45:28.432754 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.432711 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10bfbdc2-73c3-43b5-9087-bc07be95319e" containerName="cleanup" Nov 02 01:45:28.434353 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.433840 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="5779a226-eaa5-4428-99d1-47e4936591a2" containerName="cleanup" Nov 02 01:45:28.434353 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.433862 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7ebe1a8-c572-4b18-98d9-083b497aa758" containerName="cleanup" Nov 02 01:45:28.434353 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.433871 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcd48699-8090-4574-934c-b29668788243" containerName="cleanup" Nov 02 01:45:28.434353 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.433880 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="635a1467-2a08-4b8b-a8c0-b16e2a361888" containerName="cleanup" Nov 02 01:45:28.434353 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.433893 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7ebe1a8-c572-4b18-98d9-083b497aa758" containerName="cleanup" Nov 02 01:45:28.434353 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.433903 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="886fa3e6-9143-4823-903d-7e68d06100ee" containerName="cleanup" Nov 02 01:45:28.434353 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.433913 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7ebe1a8-c572-4b18-98d9-083b497aa758" containerName="cleanup" Nov 02 01:45:28.434353 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.433924 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="747eccfe-eecb-4cec-99a9-aea7857e1a34" containerName="cleanup" Nov 02 01:45:28.434353 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.433933 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="747eccfe-eecb-4cec-99a9-aea7857e1a34" containerName="cleanup" Nov 02 01:45:28.434353 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.433942 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="10bfbdc2-73c3-43b5-9087-bc07be95319e" containerName="cleanup" Nov 02 01:45:28.434353 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.433952 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3730ee2-6ba0-4dec-b103-214881818132" containerName="istio-proxy" Nov 02 01:45:28.434353 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.433961 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="10bfbdc2-73c3-43b5-9087-bc07be95319e" containerName="cleanup" Nov 02 01:45:28.434353 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.433972 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="10bfbdc2-73c3-43b5-9087-bc07be95319e" containerName="cleanup" Nov 02 01:45:28.434353 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.433987 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="635a1467-2a08-4b8b-a8c0-b16e2a361888" containerName="cleanup" Nov 02 01:45:28.434353 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.433996 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="5779a226-eaa5-4428-99d1-47e4936591a2" containerName="cleanup" Nov 02 01:45:28.434353 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.434007 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="d983a998-461c-4341-aea9-1eb65375bbb7" containerName="cleanup" Nov 02 01:45:28.434353 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.434016 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="af735a61-695b-4a66-a7cb-d0df06b3b58b" containerName="istio-proxy" Nov 02 01:45:28.434353 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.434027 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcd48699-8090-4574-934c-b29668788243" containerName="cleanup" Nov 02 01:45:28.434353 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.434038 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="d983a998-461c-4341-aea9-1eb65375bbb7" containerName="cleanup" Nov 02 01:45:28.434353 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.434052 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="886fa3e6-9143-4823-903d-7e68d06100ee" containerName="cleanup" Nov 02 01:45:28.443475 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.441439 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht"] Nov 02 01:45:28.443475 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.441495 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht" Nov 02 01:45:28.447209 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.447181 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-677312-openshift-default-dockercfg-q9hkc\"" Nov 02 01:45:28.555711 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.555677 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce4186ac-472d-4f1a-999f-7a95cffd6a05-workload-certs\") pod \"e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht\" (UID: \"ce4186ac-472d-4f1a-999f-7a95cffd6a05\") " pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht" Nov 02 01:45:28.555883 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.555779 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce4186ac-472d-4f1a-999f-7a95cffd6a05-credential-socket\") pod \"e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht\" (UID: \"ce4186ac-472d-4f1a-999f-7a95cffd6a05\") " pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht" Nov 02 01:45:28.555883 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.555814 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce4186ac-472d-4f1a-999f-7a95cffd6a05-istio-envoy\") pod \"e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht\" (UID: \"ce4186ac-472d-4f1a-999f-7a95cffd6a05\") " pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht" Nov 02 01:45:28.555883 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.555871 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce4186ac-472d-4f1a-999f-7a95cffd6a05-workload-socket\") pod \"e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht\" (UID: \"ce4186ac-472d-4f1a-999f-7a95cffd6a05\") " pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht" Nov 02 01:45:28.556038 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.556024 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce4186ac-472d-4f1a-999f-7a95cffd6a05-istio-data\") pod \"e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht\" (UID: \"ce4186ac-472d-4f1a-999f-7a95cffd6a05\") " pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht" Nov 02 01:45:28.556151 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.556134 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce4186ac-472d-4f1a-999f-7a95cffd6a05-istio-token\") pod \"e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht\" (UID: \"ce4186ac-472d-4f1a-999f-7a95cffd6a05\") " pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht" Nov 02 01:45:28.556187 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.556167 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce4186ac-472d-4f1a-999f-7a95cffd6a05-istio-podinfo\") pod \"e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht\" (UID: \"ce4186ac-472d-4f1a-999f-7a95cffd6a05\") " pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht" Nov 02 01:45:28.556245 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.556224 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce4186ac-472d-4f1a-999f-7a95cffd6a05-istiod-ca-cert\") pod \"e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht\" (UID: \"ce4186ac-472d-4f1a-999f-7a95cffd6a05\") " pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht" Nov 02 01:45:28.556287 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.556265 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndrps\" (UniqueName: \"kubernetes.io/projected/ce4186ac-472d-4f1a-999f-7a95cffd6a05-kube-api-access-ndrps\") pod \"e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht\" (UID: \"ce4186ac-472d-4f1a-999f-7a95cffd6a05\") " pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht" Nov 02 01:45:28.657847 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.657808 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce4186ac-472d-4f1a-999f-7a95cffd6a05-istio-data\") pod \"e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht\" (UID: \"ce4186ac-472d-4f1a-999f-7a95cffd6a05\") " pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht" Nov 02 01:45:28.658033 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.657903 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce4186ac-472d-4f1a-999f-7a95cffd6a05-istio-token\") pod \"e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht\" (UID: \"ce4186ac-472d-4f1a-999f-7a95cffd6a05\") " pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht" Nov 02 01:45:28.658033 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.657927 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce4186ac-472d-4f1a-999f-7a95cffd6a05-istio-podinfo\") pod \"e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht\" (UID: \"ce4186ac-472d-4f1a-999f-7a95cffd6a05\") " pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht" Nov 02 01:45:28.658033 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.657943 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce4186ac-472d-4f1a-999f-7a95cffd6a05-istiod-ca-cert\") pod \"e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht\" (UID: \"ce4186ac-472d-4f1a-999f-7a95cffd6a05\") " pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht" Nov 02 01:45:28.658033 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.657960 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndrps\" (UniqueName: \"kubernetes.io/projected/ce4186ac-472d-4f1a-999f-7a95cffd6a05-kube-api-access-ndrps\") pod \"e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht\" (UID: \"ce4186ac-472d-4f1a-999f-7a95cffd6a05\") " pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht" Nov 02 01:45:28.658033 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.658026 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce4186ac-472d-4f1a-999f-7a95cffd6a05-workload-certs\") pod \"e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht\" (UID: \"ce4186ac-472d-4f1a-999f-7a95cffd6a05\") " pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht" Nov 02 01:45:28.658315 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.658065 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce4186ac-472d-4f1a-999f-7a95cffd6a05-credential-socket\") pod \"e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht\" (UID: \"ce4186ac-472d-4f1a-999f-7a95cffd6a05\") " pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht" Nov 02 01:45:28.658315 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.658098 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce4186ac-472d-4f1a-999f-7a95cffd6a05-istio-envoy\") pod \"e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht\" (UID: \"ce4186ac-472d-4f1a-999f-7a95cffd6a05\") " pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht" Nov 02 01:45:28.658315 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.658169 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce4186ac-472d-4f1a-999f-7a95cffd6a05-workload-socket\") pod \"e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht\" (UID: \"ce4186ac-472d-4f1a-999f-7a95cffd6a05\") " pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht" Nov 02 01:45:28.658315 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.658260 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce4186ac-472d-4f1a-999f-7a95cffd6a05-istio-data\") pod \"e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht\" (UID: \"ce4186ac-472d-4f1a-999f-7a95cffd6a05\") " pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht" Nov 02 01:45:28.658564 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.658415 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce4186ac-472d-4f1a-999f-7a95cffd6a05-workload-certs\") pod \"e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht\" (UID: \"ce4186ac-472d-4f1a-999f-7a95cffd6a05\") " pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht" Nov 02 01:45:28.658564 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.658472 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce4186ac-472d-4f1a-999f-7a95cffd6a05-credential-socket\") pod \"e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht\" (UID: \"ce4186ac-472d-4f1a-999f-7a95cffd6a05\") " pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht" Nov 02 01:45:28.658643 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.658625 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce4186ac-472d-4f1a-999f-7a95cffd6a05-workload-socket\") pod \"e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht\" (UID: \"ce4186ac-472d-4f1a-999f-7a95cffd6a05\") " pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht" Nov 02 01:45:28.658985 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.658965 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce4186ac-472d-4f1a-999f-7a95cffd6a05-istiod-ca-cert\") pod \"e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht\" (UID: \"ce4186ac-472d-4f1a-999f-7a95cffd6a05\") " pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht" Nov 02 01:45:28.660561 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.660528 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce4186ac-472d-4f1a-999f-7a95cffd6a05-istio-envoy\") pod \"e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht\" (UID: \"ce4186ac-472d-4f1a-999f-7a95cffd6a05\") " pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht" Nov 02 01:45:28.660693 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.660675 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce4186ac-472d-4f1a-999f-7a95cffd6a05-istio-podinfo\") pod \"e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht\" (UID: \"ce4186ac-472d-4f1a-999f-7a95cffd6a05\") " pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht" Nov 02 01:45:28.670821 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.670792 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce4186ac-472d-4f1a-999f-7a95cffd6a05-istio-token\") pod \"e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht\" (UID: \"ce4186ac-472d-4f1a-999f-7a95cffd6a05\") " pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht" Nov 02 01:45:28.670973 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.670951 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndrps\" (UniqueName: \"kubernetes.io/projected/ce4186ac-472d-4f1a-999f-7a95cffd6a05-kube-api-access-ndrps\") pod \"e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht\" (UID: \"ce4186ac-472d-4f1a-999f-7a95cffd6a05\") " pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht" Nov 02 01:45:28.763622 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:28.763515 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht" Nov 02 01:45:29.081547 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:29.081520 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht"] Nov 02 01:45:29.083960 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:45:29.083930 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce4186ac_472d_4f1a_999f_7a95cffd6a05.slice/crio-efb248484b9a39c9a78f6a19c2389f61d9ea87241581348a50f9fd5fc7328822 WatchSource:0}: Error finding container efb248484b9a39c9a78f6a19c2389f61d9ea87241581348a50f9fd5fc7328822: Status 404 returned error can't find the container with id efb248484b9a39c9a78f6a19c2389f61d9ea87241581348a50f9fd5fc7328822 Nov 02 01:45:29.087275 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:29.087236 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:45:29.087421 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:29.087307 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:45:29.087421 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:29.087336 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:45:29.443580 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:29.443540 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht" event={"ID":"ce4186ac-472d-4f1a-999f-7a95cffd6a05","Type":"ContainerStarted","Data":"7170332b1c843f9156b8d3c9f65c0cabfcd185cd0971a000e5e5cd68e1fb084a"} Nov 02 01:45:29.444064 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:29.443608 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht" event={"ID":"ce4186ac-472d-4f1a-999f-7a95cffd6a05","Type":"ContainerStarted","Data":"efb248484b9a39c9a78f6a19c2389f61d9ea87241581348a50f9fd5fc7328822"} Nov 02 01:45:29.473535 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:29.473472 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht" podStartSLOduration=1.473451867 podStartE2EDuration="1.473451867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:45:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:45:29.470630583 +0000 UTC m=+1834.091635794" watchObservedRunningTime="2026-11-02 01:45:29.473451867 +0000 UTC m=+1834.094457081" Nov 02 01:45:29.764289 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:29.764206 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht" Nov 02 01:45:30.515027 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:30.514972 4427 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 01:45:30.515552 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:30.515061 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj" podUID="36cac416-8352-4653-8ec9-7ad64348afa7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 01:45:30.801216 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:30.800491 4427 patch_prober.go:29] interesting pod/e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 02 01:45:30.801216 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:30.800628 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht" podUID="ce4186ac-472d-4f1a-999f-7a95cffd6a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 02 01:45:31.027608 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:31.027567 4427 patch_prober.go:29] interesting pod/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 01:45:31.027898 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:31.027870 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd" podUID="d9f64fd6-8cb1-4a34-99f0-828c753a1119" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 01:45:31.611818 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:45:31.611771 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:45:31.766399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:31.764784 4427 patch_prober.go:29] interesting pod/e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 02 01:45:31.766399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:31.764847 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht" podUID="ce4186ac-472d-4f1a-999f-7a95cffd6a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 02 01:45:32.769399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:32.767706 4427 patch_prober.go:29] interesting pod/e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 02 01:45:32.769399 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:32.767762 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht" podUID="ce4186ac-472d-4f1a-999f-7a95cffd6a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 02 01:45:33.529439 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.529403 4427 generic.go:356] "Generic (PLEG): container finished" podID="36cac416-8352-4653-8ec9-7ad64348afa7" containerID="cf5643a4140429f936a509f8ea84de25169cef9edca00cf73746c2659bd2a183" exitCode=0 Nov 02 01:45:33.533177 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.533041 4427 generic.go:356] "Generic (PLEG): container finished" podID="82d82a81-f1df-446d-b07b-20f510354a7e" containerID="1f7f09782a19394d9cf33bb0bd9e856f43155b286d13912c18e7d0b10098df76" exitCode=0 Nov 02 01:45:33.538981 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.538853 4427 generic.go:356] "Generic (PLEG): container finished" podID="d9f64fd6-8cb1-4a34-99f0-828c753a1119" containerID="ba3e2481997f8ddb323e8b6862e4cf4a4bd794e8f35a8e774e0aae11265a8a5a" exitCode=0 Nov 02 01:45:33.541141 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.540793 4427 generic.go:356] "Generic (PLEG): container finished" podID="136eabea-7a63-4d89-bcc1-0e3d6c8810cf" containerID="e7bad34c068724ee09f318ec5119e242811f511e6d11437373e7b874c693f60a" exitCode=0 Nov 02 01:45:33.608432 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.605556 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj" event={"ID":"36cac416-8352-4653-8ec9-7ad64348afa7","Type":"ContainerDied","Data":"cf5643a4140429f936a509f8ea84de25169cef9edca00cf73746c2659bd2a183"} Nov 02 01:45:33.608432 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.605638 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt" event={"ID":"82d82a81-f1df-446d-b07b-20f510354a7e","Type":"ContainerDied","Data":"1f7f09782a19394d9cf33bb0bd9e856f43155b286d13912c18e7d0b10098df76"} Nov 02 01:45:33.608432 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.605674 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd" event={"ID":"d9f64fd6-8cb1-4a34-99f0-828c753a1119","Type":"ContainerDied","Data":"ba3e2481997f8ddb323e8b6862e4cf4a4bd794e8f35a8e774e0aae11265a8a5a"} Nov 02 01:45:33.608432 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.605702 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm" event={"ID":"136eabea-7a63-4d89-bcc1-0e3d6c8810cf","Type":"ContainerDied","Data":"e7bad34c068724ee09f318ec5119e242811f511e6d11437373e7b874c693f60a"} Nov 02 01:45:33.701195 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.701165 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd" Nov 02 01:45:33.746017 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.745555 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm" Nov 02 01:45:33.799457 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.798136 4427 patch_prober.go:29] interesting pod/e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 02 01:45:33.799457 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.798214 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht" podUID="ce4186ac-472d-4f1a-999f-7a95cffd6a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 02 01:45:33.803205 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.803173 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/136eabea-7a63-4d89-bcc1-0e3d6c8810cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/136eabea-7a63-4d89-bcc1-0e3d6c8810cf-workload-certs\") pod \"136eabea-7a63-4d89-bcc1-0e3d6c8810cf\" (UID: \"136eabea-7a63-4d89-bcc1-0e3d6c8810cf\") " Nov 02 01:45:33.803356 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.803233 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9f64fd6-8cb1-4a34-99f0-828c753a1119-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9f64fd6-8cb1-4a34-99f0-828c753a1119-istio-envoy\") pod \"d9f64fd6-8cb1-4a34-99f0-828c753a1119\" (UID: \"d9f64fd6-8cb1-4a34-99f0-828c753a1119\") " Nov 02 01:45:33.803356 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.803262 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/136eabea-7a63-4d89-bcc1-0e3d6c8810cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/136eabea-7a63-4d89-bcc1-0e3d6c8810cf-workload-socket\") pod \"136eabea-7a63-4d89-bcc1-0e3d6c8810cf\" (UID: \"136eabea-7a63-4d89-bcc1-0e3d6c8810cf\") " Nov 02 01:45:33.803496 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.803445 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/136eabea-7a63-4d89-bcc1-0e3d6c8810cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/136eabea-7a63-4d89-bcc1-0e3d6c8810cf-istio-podinfo\") pod \"136eabea-7a63-4d89-bcc1-0e3d6c8810cf\" (UID: \"136eabea-7a63-4d89-bcc1-0e3d6c8810cf\") " Nov 02 01:45:33.803496 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.803482 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/136eabea-7a63-4d89-bcc1-0e3d6c8810cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/136eabea-7a63-4d89-bcc1-0e3d6c8810cf-istiod-ca-cert\") pod \"136eabea-7a63-4d89-bcc1-0e3d6c8810cf\" (UID: \"136eabea-7a63-4d89-bcc1-0e3d6c8810cf\") " Nov 02 01:45:33.803572 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.803549 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/136eabea-7a63-4d89-bcc1-0e3d6c8810cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/136eabea-7a63-4d89-bcc1-0e3d6c8810cf-istio-envoy\") pod \"136eabea-7a63-4d89-bcc1-0e3d6c8810cf\" (UID: \"136eabea-7a63-4d89-bcc1-0e3d6c8810cf\") " Nov 02 01:45:33.803942 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.803914 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/136eabea-7a63-4d89-bcc1-0e3d6c8810cf-workload-certs" pod "136eabea-7a63-4d89-bcc1-0e3d6c8810cf" (UID: "136eabea-7a63-4d89-bcc1-0e3d6c8810cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:33.804145 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.804126 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/136eabea-7a63-4d89-bcc1-0e3d6c8810cf-workload-socket" pod "136eabea-7a63-4d89-bcc1-0e3d6c8810cf" (UID: "136eabea-7a63-4d89-bcc1-0e3d6c8810cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:33.804507 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.804485 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/136eabea-7a63-4d89-bcc1-0e3d6c8810cf-istiod-ca-cert" pod "136eabea-7a63-4d89-bcc1-0e3d6c8810cf" (UID: "136eabea-7a63-4d89-bcc1-0e3d6c8810cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:45:33.804660 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.804640 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9f64fd6-8cb1-4a34-99f0-828c753a1119-istio-token\" (UniqueName: \"kubernetes.io/projected/d9f64fd6-8cb1-4a34-99f0-828c753a1119-istio-token\") pod \"d9f64fd6-8cb1-4a34-99f0-828c753a1119\" (UID: \"d9f64fd6-8cb1-4a34-99f0-828c753a1119\") " Nov 02 01:45:33.804721 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.804682 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/136eabea-7a63-4d89-bcc1-0e3d6c8810cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/136eabea-7a63-4d89-bcc1-0e3d6c8810cf-istio-data\") pod \"136eabea-7a63-4d89-bcc1-0e3d6c8810cf\" (UID: \"136eabea-7a63-4d89-bcc1-0e3d6c8810cf\") " Nov 02 01:45:33.804756 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.804717 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/136eabea-7a63-4d89-bcc1-0e3d6c8810cf-istio-token\" (UniqueName: \"kubernetes.io/projected/136eabea-7a63-4d89-bcc1-0e3d6c8810cf-istio-token\") pod \"136eabea-7a63-4d89-bcc1-0e3d6c8810cf\" (UID: \"136eabea-7a63-4d89-bcc1-0e3d6c8810cf\") " Nov 02 01:45:33.804805 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.804790 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/136eabea-7a63-4d89-bcc1-0e3d6c8810cf-kube-api-access-tkhcn\" (UniqueName: \"kubernetes.io/projected/136eabea-7a63-4d89-bcc1-0e3d6c8810cf-kube-api-access-tkhcn\") pod \"136eabea-7a63-4d89-bcc1-0e3d6c8810cf\" (UID: \"136eabea-7a63-4d89-bcc1-0e3d6c8810cf\") " Nov 02 01:45:33.804862 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.804848 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d9f64fd6-8cb1-4a34-99f0-828c753a1119-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9f64fd6-8cb1-4a34-99f0-828c753a1119-istio-podinfo\") pod \"d9f64fd6-8cb1-4a34-99f0-828c753a1119\" (UID: \"d9f64fd6-8cb1-4a34-99f0-828c753a1119\") " Nov 02 01:45:33.804935 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.804922 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9f64fd6-8cb1-4a34-99f0-828c753a1119-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9f64fd6-8cb1-4a34-99f0-828c753a1119-istio-data\") pod \"d9f64fd6-8cb1-4a34-99f0-828c753a1119\" (UID: \"d9f64fd6-8cb1-4a34-99f0-828c753a1119\") " Nov 02 01:45:33.805649 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.805283 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/136eabea-7a63-4d89-bcc1-0e3d6c8810cf-istio-data" pod "136eabea-7a63-4d89-bcc1-0e3d6c8810cf" (UID: "136eabea-7a63-4d89-bcc1-0e3d6c8810cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:33.806029 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.805899 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9f64fd6-8cb1-4a34-99f0-828c753a1119-istio-data" pod "d9f64fd6-8cb1-4a34-99f0-828c753a1119" (UID: "d9f64fd6-8cb1-4a34-99f0-828c753a1119"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:33.806260 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.806237 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9f64fd6-8cb1-4a34-99f0-828c753a1119-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9f64fd6-8cb1-4a34-99f0-828c753a1119-workload-certs\") pod \"d9f64fd6-8cb1-4a34-99f0-828c753a1119\" (UID: \"d9f64fd6-8cb1-4a34-99f0-828c753a1119\") " Nov 02 01:45:33.807334 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.806445 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/136eabea-7a63-4d89-bcc1-0e3d6c8810cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/136eabea-7a63-4d89-bcc1-0e3d6c8810cf-credential-socket\") pod \"136eabea-7a63-4d89-bcc1-0e3d6c8810cf\" (UID: \"136eabea-7a63-4d89-bcc1-0e3d6c8810cf\") " Nov 02 01:45:33.807334 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.806524 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9f64fd6-8cb1-4a34-99f0-828c753a1119-workload-certs" pod "d9f64fd6-8cb1-4a34-99f0-828c753a1119" (UID: "d9f64fd6-8cb1-4a34-99f0-828c753a1119"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:33.807334 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.806702 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/136eabea-7a63-4d89-bcc1-0e3d6c8810cf-credential-socket" pod "136eabea-7a63-4d89-bcc1-0e3d6c8810cf" (UID: "136eabea-7a63-4d89-bcc1-0e3d6c8810cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:33.807334 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.806810 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9f64fd6-8cb1-4a34-99f0-828c753a1119-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9f64fd6-8cb1-4a34-99f0-828c753a1119-workload-socket\") pod \"d9f64fd6-8cb1-4a34-99f0-828c753a1119\" (UID: \"d9f64fd6-8cb1-4a34-99f0-828c753a1119\") " Nov 02 01:45:33.807334 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.806875 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9f64fd6-8cb1-4a34-99f0-828c753a1119-kube-api-access-xrfqc\" (UniqueName: \"kubernetes.io/projected/d9f64fd6-8cb1-4a34-99f0-828c753a1119-kube-api-access-xrfqc\") pod \"d9f64fd6-8cb1-4a34-99f0-828c753a1119\" (UID: \"d9f64fd6-8cb1-4a34-99f0-828c753a1119\") " Nov 02 01:45:33.807921 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.807342 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9f64fd6-8cb1-4a34-99f0-828c753a1119-credential-socket" pod "d9f64fd6-8cb1-4a34-99f0-828c753a1119" (UID: "d9f64fd6-8cb1-4a34-99f0-828c753a1119"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:33.807921 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.807643 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/136eabea-7a63-4d89-bcc1-0e3d6c8810cf-istio-envoy" pod "136eabea-7a63-4d89-bcc1-0e3d6c8810cf" (UID: "136eabea-7a63-4d89-bcc1-0e3d6c8810cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:33.807921 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.807693 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9f64fd6-8cb1-4a34-99f0-828c753a1119-istio-envoy" pod "d9f64fd6-8cb1-4a34-99f0-828c753a1119" (UID: "d9f64fd6-8cb1-4a34-99f0-828c753a1119"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:33.807921 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.807779 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9f64fd6-8cb1-4a34-99f0-828c753a1119-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9f64fd6-8cb1-4a34-99f0-828c753a1119-credential-socket\") pod \"d9f64fd6-8cb1-4a34-99f0-828c753a1119\" (UID: \"d9f64fd6-8cb1-4a34-99f0-828c753a1119\") " Nov 02 01:45:33.807921 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.807870 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9f64fd6-8cb1-4a34-99f0-828c753a1119-workload-socket" pod "d9f64fd6-8cb1-4a34-99f0-828c753a1119" (UID: "d9f64fd6-8cb1-4a34-99f0-828c753a1119"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:33.808172 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.808082 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9f64fd6-8cb1-4a34-99f0-828c753a1119-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9f64fd6-8cb1-4a34-99f0-828c753a1119-istiod-ca-cert\") pod \"d9f64fd6-8cb1-4a34-99f0-828c753a1119\" (UID: \"d9f64fd6-8cb1-4a34-99f0-828c753a1119\") " Nov 02 01:45:33.809662 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.808594 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9f64fd6-8cb1-4a34-99f0-828c753a1119-istiod-ca-cert" pod "d9f64fd6-8cb1-4a34-99f0-828c753a1119" (UID: "d9f64fd6-8cb1-4a34-99f0-828c753a1119"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:45:33.809662 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.808976 4427 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/136eabea-7a63-4d89-bcc1-0e3d6c8810cf-workload-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:33.809662 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.809001 4427 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9f64fd6-8cb1-4a34-99f0-828c753a1119-istio-envoy\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:33.809662 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.809015 4427 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/136eabea-7a63-4d89-bcc1-0e3d6c8810cf-workload-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:33.809662 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.809030 4427 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/136eabea-7a63-4d89-bcc1-0e3d6c8810cf-istiod-ca-cert\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:33.809662 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.809045 4427 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/136eabea-7a63-4d89-bcc1-0e3d6c8810cf-istio-envoy\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:33.809662 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.809059 4427 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/136eabea-7a63-4d89-bcc1-0e3d6c8810cf-istio-data\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:33.809662 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.809073 4427 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9f64fd6-8cb1-4a34-99f0-828c753a1119-istio-data\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:33.809662 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.809086 4427 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9f64fd6-8cb1-4a34-99f0-828c753a1119-workload-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:33.809662 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.809100 4427 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/136eabea-7a63-4d89-bcc1-0e3d6c8810cf-credential-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:33.809662 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.809114 4427 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9f64fd6-8cb1-4a34-99f0-828c753a1119-workload-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:33.809662 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.809128 4427 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9f64fd6-8cb1-4a34-99f0-828c753a1119-credential-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:33.809662 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.809144 4427 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9f64fd6-8cb1-4a34-99f0-828c753a1119-istiod-ca-cert\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:33.811721 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.811684 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9f64fd6-8cb1-4a34-99f0-828c753a1119-istio-token" pod "d9f64fd6-8cb1-4a34-99f0-828c753a1119" (UID: "d9f64fd6-8cb1-4a34-99f0-828c753a1119"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:45:33.812289 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.812263 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/136eabea-7a63-4d89-bcc1-0e3d6c8810cf-istio-podinfo" pod "136eabea-7a63-4d89-bcc1-0e3d6c8810cf" (UID: "136eabea-7a63-4d89-bcc1-0e3d6c8810cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:45:33.812733 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.812707 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d9f64fd6-8cb1-4a34-99f0-828c753a1119-istio-podinfo" pod "d9f64fd6-8cb1-4a34-99f0-828c753a1119" (UID: "d9f64fd6-8cb1-4a34-99f0-828c753a1119"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:45:33.813070 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.812939 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/136eabea-7a63-4d89-bcc1-0e3d6c8810cf-kube-api-access-tkhcn" pod "136eabea-7a63-4d89-bcc1-0e3d6c8810cf" (UID: "136eabea-7a63-4d89-bcc1-0e3d6c8810cf"). InnerVolumeSpecName "kube-api-access-tkhcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:45:33.813638 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.813611 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/136eabea-7a63-4d89-bcc1-0e3d6c8810cf-istio-token" pod "136eabea-7a63-4d89-bcc1-0e3d6c8810cf" (UID: "136eabea-7a63-4d89-bcc1-0e3d6c8810cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:45:33.815607 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.815581 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9f64fd6-8cb1-4a34-99f0-828c753a1119-kube-api-access-xrfqc" pod "d9f64fd6-8cb1-4a34-99f0-828c753a1119" (UID: "d9f64fd6-8cb1-4a34-99f0-828c753a1119"). InnerVolumeSpecName "kube-api-access-xrfqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:45:33.833981 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.833721 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj" Nov 02 01:45:33.910558 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.910513 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36cac416-8352-4653-8ec9-7ad64348afa7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36cac416-8352-4653-8ec9-7ad64348afa7-istio-data\") pod \"36cac416-8352-4653-8ec9-7ad64348afa7\" (UID: \"36cac416-8352-4653-8ec9-7ad64348afa7\") " Nov 02 01:45:33.910818 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.910610 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36cac416-8352-4653-8ec9-7ad64348afa7-kube-api-access-725hj\" (UniqueName: \"kubernetes.io/projected/36cac416-8352-4653-8ec9-7ad64348afa7-kube-api-access-725hj\") pod \"36cac416-8352-4653-8ec9-7ad64348afa7\" (UID: \"36cac416-8352-4653-8ec9-7ad64348afa7\") " Nov 02 01:45:33.910818 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.910654 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36cac416-8352-4653-8ec9-7ad64348afa7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36cac416-8352-4653-8ec9-7ad64348afa7-istiod-ca-cert\") pod \"36cac416-8352-4653-8ec9-7ad64348afa7\" (UID: \"36cac416-8352-4653-8ec9-7ad64348afa7\") " Nov 02 01:45:33.910818 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.910691 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/36cac416-8352-4653-8ec9-7ad64348afa7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36cac416-8352-4653-8ec9-7ad64348afa7-istio-podinfo\") pod \"36cac416-8352-4653-8ec9-7ad64348afa7\" (UID: \"36cac416-8352-4653-8ec9-7ad64348afa7\") " Nov 02 01:45:33.910818 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.910742 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36cac416-8352-4653-8ec9-7ad64348afa7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36cac416-8352-4653-8ec9-7ad64348afa7-credential-socket\") pod \"36cac416-8352-4653-8ec9-7ad64348afa7\" (UID: \"36cac416-8352-4653-8ec9-7ad64348afa7\") " Nov 02 01:45:33.911097 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.910852 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36cac416-8352-4653-8ec9-7ad64348afa7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36cac416-8352-4653-8ec9-7ad64348afa7-istio-envoy\") pod \"36cac416-8352-4653-8ec9-7ad64348afa7\" (UID: \"36cac416-8352-4653-8ec9-7ad64348afa7\") " Nov 02 01:45:33.911097 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.911011 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36cac416-8352-4653-8ec9-7ad64348afa7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36cac416-8352-4653-8ec9-7ad64348afa7-workload-socket\") pod \"36cac416-8352-4653-8ec9-7ad64348afa7\" (UID: \"36cac416-8352-4653-8ec9-7ad64348afa7\") " Nov 02 01:45:33.911299 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.911121 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36cac416-8352-4653-8ec9-7ad64348afa7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36cac416-8352-4653-8ec9-7ad64348afa7-workload-certs\") pod \"36cac416-8352-4653-8ec9-7ad64348afa7\" (UID: \"36cac416-8352-4653-8ec9-7ad64348afa7\") " Nov 02 01:45:33.911299 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.911158 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36cac416-8352-4653-8ec9-7ad64348afa7-istio-token\" (UniqueName: \"kubernetes.io/projected/36cac416-8352-4653-8ec9-7ad64348afa7-istio-token\") pod \"36cac416-8352-4653-8ec9-7ad64348afa7\" (UID: \"36cac416-8352-4653-8ec9-7ad64348afa7\") " Nov 02 01:45:33.912748 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.912719 4427 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9f64fd6-8cb1-4a34-99f0-828c753a1119-istio-token\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:33.912909 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.912894 4427 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/136eabea-7a63-4d89-bcc1-0e3d6c8810cf-istio-token\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:33.913010 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.912994 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkhcn\" (UniqueName: \"kubernetes.io/projected/136eabea-7a63-4d89-bcc1-0e3d6c8810cf-kube-api-access-tkhcn\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:33.913104 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.913091 4427 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9f64fd6-8cb1-4a34-99f0-828c753a1119-istio-podinfo\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:33.913325 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.913269 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrfqc\" (UniqueName: \"kubernetes.io/projected/d9f64fd6-8cb1-4a34-99f0-828c753a1119-kube-api-access-xrfqc\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:33.913518 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.913504 4427 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/136eabea-7a63-4d89-bcc1-0e3d6c8810cf-istio-podinfo\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:33.913622 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.913173 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36cac416-8352-4653-8ec9-7ad64348afa7-credential-socket" pod "36cac416-8352-4653-8ec9-7ad64348afa7" (UID: "36cac416-8352-4653-8ec9-7ad64348afa7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:33.913724 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.913431 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36cac416-8352-4653-8ec9-7ad64348afa7-workload-certs" pod "36cac416-8352-4653-8ec9-7ad64348afa7" (UID: "36cac416-8352-4653-8ec9-7ad64348afa7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:33.913813 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.913445 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36cac416-8352-4653-8ec9-7ad64348afa7-istio-data" pod "36cac416-8352-4653-8ec9-7ad64348afa7" (UID: "36cac416-8352-4653-8ec9-7ad64348afa7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:33.913911 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.913492 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36cac416-8352-4653-8ec9-7ad64348afa7-workload-socket" pod "36cac416-8352-4653-8ec9-7ad64348afa7" (UID: "36cac416-8352-4653-8ec9-7ad64348afa7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:33.914034 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.913623 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36cac416-8352-4653-8ec9-7ad64348afa7-istiod-ca-cert" pod "36cac416-8352-4653-8ec9-7ad64348afa7" (UID: "36cac416-8352-4653-8ec9-7ad64348afa7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:45:33.916345 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.916299 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36cac416-8352-4653-8ec9-7ad64348afa7-istio-token" pod "36cac416-8352-4653-8ec9-7ad64348afa7" (UID: "36cac416-8352-4653-8ec9-7ad64348afa7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:45:33.918252 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.918217 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36cac416-8352-4653-8ec9-7ad64348afa7-kube-api-access-725hj" pod "36cac416-8352-4653-8ec9-7ad64348afa7" (UID: "36cac416-8352-4653-8ec9-7ad64348afa7"). InnerVolumeSpecName "kube-api-access-725hj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:45:33.919245 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.919210 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36cac416-8352-4653-8ec9-7ad64348afa7-istio-envoy" pod "36cac416-8352-4653-8ec9-7ad64348afa7" (UID: "36cac416-8352-4653-8ec9-7ad64348afa7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:33.919728 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:33.919695 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/36cac416-8352-4653-8ec9-7ad64348afa7-istio-podinfo" pod "36cac416-8352-4653-8ec9-7ad64348afa7" (UID: "36cac416-8352-4653-8ec9-7ad64348afa7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:45:34.014977 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:34.014935 4427 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36cac416-8352-4653-8ec9-7ad64348afa7-istio-envoy\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:34.014977 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:34.014981 4427 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36cac416-8352-4653-8ec9-7ad64348afa7-workload-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:34.015189 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:34.015006 4427 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36cac416-8352-4653-8ec9-7ad64348afa7-workload-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:34.015189 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:34.015022 4427 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36cac416-8352-4653-8ec9-7ad64348afa7-istio-token\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:34.015189 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:34.015035 4427 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36cac416-8352-4653-8ec9-7ad64348afa7-istio-data\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:34.015189 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:34.015048 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-725hj\" (UniqueName: \"kubernetes.io/projected/36cac416-8352-4653-8ec9-7ad64348afa7-kube-api-access-725hj\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:34.015189 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:34.015066 4427 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36cac416-8352-4653-8ec9-7ad64348afa7-istiod-ca-cert\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:34.015189 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:34.015079 4427 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36cac416-8352-4653-8ec9-7ad64348afa7-istio-podinfo\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:34.015189 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:34.015092 4427 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36cac416-8352-4653-8ec9-7ad64348afa7-credential-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:34.056809 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:34.056736 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt" Nov 02 01:45:34.116876 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:34.116837 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82d82a81-f1df-446d-b07b-20f510354a7e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82d82a81-f1df-446d-b07b-20f510354a7e-credential-socket\") pod \"82d82a81-f1df-446d-b07b-20f510354a7e\" (UID: \"82d82a81-f1df-446d-b07b-20f510354a7e\") " Nov 02 01:45:34.117068 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:34.116970 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/82d82a81-f1df-446d-b07b-20f510354a7e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82d82a81-f1df-446d-b07b-20f510354a7e-istio-podinfo\") pod \"82d82a81-f1df-446d-b07b-20f510354a7e\" (UID: \"82d82a81-f1df-446d-b07b-20f510354a7e\") " Nov 02 01:45:34.117165 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:34.117094 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82d82a81-f1df-446d-b07b-20f510354a7e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82d82a81-f1df-446d-b07b-20f510354a7e-istiod-ca-cert\") pod \"82d82a81-f1df-446d-b07b-20f510354a7e\" (UID: \"82d82a81-f1df-446d-b07b-20f510354a7e\") " Nov 02 01:45:34.117165 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:34.117129 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82d82a81-f1df-446d-b07b-20f510354a7e-credential-socket" pod "82d82a81-f1df-446d-b07b-20f510354a7e" (UID: "82d82a81-f1df-446d-b07b-20f510354a7e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:34.117165 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:34.117141 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82d82a81-f1df-446d-b07b-20f510354a7e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82d82a81-f1df-446d-b07b-20f510354a7e-workload-certs\") pod \"82d82a81-f1df-446d-b07b-20f510354a7e\" (UID: \"82d82a81-f1df-446d-b07b-20f510354a7e\") " Nov 02 01:45:34.117334 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:34.117169 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82d82a81-f1df-446d-b07b-20f510354a7e-istio-token\" (UniqueName: \"kubernetes.io/projected/82d82a81-f1df-446d-b07b-20f510354a7e-istio-token\") pod \"82d82a81-f1df-446d-b07b-20f510354a7e\" (UID: \"82d82a81-f1df-446d-b07b-20f510354a7e\") " Nov 02 01:45:34.117334 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:34.117249 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82d82a81-f1df-446d-b07b-20f510354a7e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82d82a81-f1df-446d-b07b-20f510354a7e-workload-socket\") pod \"82d82a81-f1df-446d-b07b-20f510354a7e\" (UID: \"82d82a81-f1df-446d-b07b-20f510354a7e\") " Nov 02 01:45:34.117334 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:34.117305 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82d82a81-f1df-446d-b07b-20f510354a7e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82d82a81-f1df-446d-b07b-20f510354a7e-istio-data\") pod \"82d82a81-f1df-446d-b07b-20f510354a7e\" (UID: \"82d82a81-f1df-446d-b07b-20f510354a7e\") " Nov 02 01:45:34.117551 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:34.117391 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82d82a81-f1df-446d-b07b-20f510354a7e-kube-api-access-7mzz6\" (UniqueName: \"kubernetes.io/projected/82d82a81-f1df-446d-b07b-20f510354a7e-kube-api-access-7mzz6\") pod \"82d82a81-f1df-446d-b07b-20f510354a7e\" (UID: \"82d82a81-f1df-446d-b07b-20f510354a7e\") " Nov 02 01:45:34.117551 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:34.117423 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82d82a81-f1df-446d-b07b-20f510354a7e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82d82a81-f1df-446d-b07b-20f510354a7e-istio-envoy\") pod \"82d82a81-f1df-446d-b07b-20f510354a7e\" (UID: \"82d82a81-f1df-446d-b07b-20f510354a7e\") " Nov 02 01:45:34.117551 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:34.117434 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82d82a81-f1df-446d-b07b-20f510354a7e-istiod-ca-cert" pod "82d82a81-f1df-446d-b07b-20f510354a7e" (UID: "82d82a81-f1df-446d-b07b-20f510354a7e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:45:34.117720 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:34.117611 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82d82a81-f1df-446d-b07b-20f510354a7e-workload-socket" pod "82d82a81-f1df-446d-b07b-20f510354a7e" (UID: "82d82a81-f1df-446d-b07b-20f510354a7e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:34.117720 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:34.117712 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82d82a81-f1df-446d-b07b-20f510354a7e-istio-data" pod "82d82a81-f1df-446d-b07b-20f510354a7e" (UID: "82d82a81-f1df-446d-b07b-20f510354a7e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:34.117936 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:34.117914 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82d82a81-f1df-446d-b07b-20f510354a7e-workload-certs" pod "82d82a81-f1df-446d-b07b-20f510354a7e" (UID: "82d82a81-f1df-446d-b07b-20f510354a7e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:34.118523 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:34.118497 4427 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82d82a81-f1df-446d-b07b-20f510354a7e-workload-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:34.118629 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:34.118527 4427 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82d82a81-f1df-446d-b07b-20f510354a7e-istio-data\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:34.118629 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:34.118542 4427 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82d82a81-f1df-446d-b07b-20f510354a7e-credential-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:34.118629 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:34.118555 4427 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82d82a81-f1df-446d-b07b-20f510354a7e-istiod-ca-cert\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:34.118629 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:34.118568 4427 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82d82a81-f1df-446d-b07b-20f510354a7e-workload-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:34.120624 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:34.120603 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82d82a81-f1df-446d-b07b-20f510354a7e-kube-api-access-7mzz6" pod "82d82a81-f1df-446d-b07b-20f510354a7e" (UID: "82d82a81-f1df-446d-b07b-20f510354a7e"). InnerVolumeSpecName "kube-api-access-7mzz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:45:34.120624 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:34.120609 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82d82a81-f1df-446d-b07b-20f510354a7e-istio-envoy" pod "82d82a81-f1df-446d-b07b-20f510354a7e" (UID: "82d82a81-f1df-446d-b07b-20f510354a7e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:34.120771 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:34.120692 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82d82a81-f1df-446d-b07b-20f510354a7e-istio-podinfo" pod "82d82a81-f1df-446d-b07b-20f510354a7e" (UID: "82d82a81-f1df-446d-b07b-20f510354a7e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:45:34.121514 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:34.121493 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82d82a81-f1df-446d-b07b-20f510354a7e-istio-token" pod "82d82a81-f1df-446d-b07b-20f510354a7e" (UID: "82d82a81-f1df-446d-b07b-20f510354a7e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:45:34.220162 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:34.220119 4427 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82d82a81-f1df-446d-b07b-20f510354a7e-istio-podinfo\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:34.220162 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:34.220152 4427 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82d82a81-f1df-446d-b07b-20f510354a7e-istio-token\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:34.220162 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:34.220163 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mzz6\" (UniqueName: \"kubernetes.io/projected/82d82a81-f1df-446d-b07b-20f510354a7e-kube-api-access-7mzz6\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:34.220386 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:34.220174 4427 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82d82a81-f1df-446d-b07b-20f510354a7e-istio-envoy\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:34.550690 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:34.550659 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd" Nov 02 01:45:34.550901 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:34.550697 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd" event={"ID":"d9f64fd6-8cb1-4a34-99f0-828c753a1119","Type":"ContainerDied","Data":"75748979247eec2a3f049298fc50eff90c26aa01ec9939714047e889a64140e8"} Nov 02 01:45:34.550901 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:34.550761 4427 scope.go:122] "RemoveContainer" containerID="ba3e2481997f8ddb323e8b6862e4cf4a4bd794e8f35a8e774e0aae11265a8a5a" Nov 02 01:45:34.552847 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:34.552429 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm" Nov 02 01:45:34.552847 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:34.552535 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm" event={"ID":"136eabea-7a63-4d89-bcc1-0e3d6c8810cf","Type":"ContainerDied","Data":"ae0dac58e7a7f94ec3ad20c5e78f45c66b5c5d442ecbe8d670cdefdccd29d41f"} Nov 02 01:45:34.557199 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:34.557176 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj" Nov 02 01:45:34.557326 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:34.557247 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj" event={"ID":"36cac416-8352-4653-8ec9-7ad64348afa7","Type":"ContainerDied","Data":"9e0e090a19e1d65affa6fab6ccfd8a8f492bf848b1476ec9c4ca525c4b1dc602"} Nov 02 01:45:34.558886 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:34.558864 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt" event={"ID":"82d82a81-f1df-446d-b07b-20f510354a7e","Type":"ContainerDied","Data":"2e53271e26093c545760bb22ce0188096d60ab876b6883cf401dbc3a0fae75e6"} Nov 02 01:45:34.558886 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:34.558883 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt" Nov 02 01:45:34.589348 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:34.589320 4427 scope.go:122] "RemoveContainer" containerID="e7bad34c068724ee09f318ec5119e242811f511e6d11437373e7b874c693f60a" Nov 02 01:45:34.603983 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:34.603959 4427 scope.go:122] "RemoveContainer" containerID="cf5643a4140429f936a509f8ea84de25169cef9edca00cf73746c2659bd2a183" Nov 02 01:45:34.614935 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:34.614899 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd"] Nov 02 01:45:34.621590 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:34.621553 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-792nd"] Nov 02 01:45:34.623586 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:34.623404 4427 scope.go:122] "RemoveContainer" containerID="1f7f09782a19394d9cf33bb0bd9e856f43155b286d13912c18e7d0b10098df76" Nov 02 01:45:34.649842 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:34.649811 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm"] Nov 02 01:45:34.662082 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:34.659815 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-b9whm"] Nov 02 01:45:34.701285 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:34.701256 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj"] Nov 02 01:45:34.704684 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:34.704654 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-4h5lj"] Nov 02 01:45:34.710326 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:34.710298 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt"] Nov 02 01:45:34.714232 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:34.714204 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-f689bc-openshift-default-b4b8cb57c-8ksnt"] Nov 02 01:45:34.783826 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:34.783792 4427 patch_prober.go:29] interesting pod/e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 02 01:45:34.784112 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:34.784087 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht" podUID="ce4186ac-472d-4f1a-999f-7a95cffd6a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 02 01:45:35.536688 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:35.536657 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="136eabea-7a63-4d89-bcc1-0e3d6c8810cf" path="/var/lib/kubelet/pods/136eabea-7a63-4d89-bcc1-0e3d6c8810cf/volumes" Nov 02 01:45:35.537234 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:35.537218 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36cac416-8352-4653-8ec9-7ad64348afa7" path="/var/lib/kubelet/pods/36cac416-8352-4653-8ec9-7ad64348afa7/volumes" Nov 02 01:45:35.537737 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:35.537723 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82d82a81-f1df-446d-b07b-20f510354a7e" path="/var/lib/kubelet/pods/82d82a81-f1df-446d-b07b-20f510354a7e/volumes" Nov 02 01:45:35.538592 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:35.538575 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9f64fd6-8cb1-4a34-99f0-828c753a1119" path="/var/lib/kubelet/pods/d9f64fd6-8cb1-4a34-99f0-828c753a1119/volumes" Nov 02 01:45:35.557813 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:35.557751 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-12e7cb/test-model-12e7cb-kserve-5b5d7468c9-xkkqs" podUID="21ee65a5-81e6-4736-91ac-6399ba34947e" containerName="main" containerID="cri-o://9630734ee18562779f539a58dd9f4825d78c0d80937338443fb9fe92cc889f11" gracePeriod=45 Nov 02 01:45:35.777400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:35.773791 4427 patch_prober.go:29] interesting pod/e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 02 01:45:35.777400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:35.773852 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht" podUID="ce4186ac-472d-4f1a-999f-7a95cffd6a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 02 01:45:35.944796 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:35.944768 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-12e7cb/test-model-12e7cb-kserve-5b5d7468c9-xkkqs" Nov 02 01:45:36.051633 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:36.051588 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21ee65a5-81e6-4736-91ac-6399ba34947e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21ee65a5-81e6-4736-91ac-6399ba34947e-tmp-dir\") pod \"21ee65a5-81e6-4736-91ac-6399ba34947e\" (UID: \"21ee65a5-81e6-4736-91ac-6399ba34947e\") " Nov 02 01:45:36.051824 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:36.051715 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21ee65a5-81e6-4736-91ac-6399ba34947e-tls-certs\" (UniqueName: \"kubernetes.io/secret/21ee65a5-81e6-4736-91ac-6399ba34947e-tls-certs\") pod \"21ee65a5-81e6-4736-91ac-6399ba34947e\" (UID: \"21ee65a5-81e6-4736-91ac-6399ba34947e\") " Nov 02 01:45:36.051824 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:36.051754 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21ee65a5-81e6-4736-91ac-6399ba34947e-kube-api-access-vzxpt\" (UniqueName: \"kubernetes.io/projected/21ee65a5-81e6-4736-91ac-6399ba34947e-kube-api-access-vzxpt\") pod \"21ee65a5-81e6-4736-91ac-6399ba34947e\" (UID: \"21ee65a5-81e6-4736-91ac-6399ba34947e\") " Nov 02 01:45:36.051987 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:36.051840 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21ee65a5-81e6-4736-91ac-6399ba34947e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21ee65a5-81e6-4736-91ac-6399ba34947e-model-cache\") pod \"21ee65a5-81e6-4736-91ac-6399ba34947e\" (UID: \"21ee65a5-81e6-4736-91ac-6399ba34947e\") " Nov 02 01:45:36.051987 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:36.051883 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21ee65a5-81e6-4736-91ac-6399ba34947e-home\" (UniqueName: \"kubernetes.io/empty-dir/21ee65a5-81e6-4736-91ac-6399ba34947e-home\") pod \"21ee65a5-81e6-4736-91ac-6399ba34947e\" (UID: \"21ee65a5-81e6-4736-91ac-6399ba34947e\") " Nov 02 01:45:36.051987 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:36.051920 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21ee65a5-81e6-4736-91ac-6399ba34947e-tmp-dir" pod "21ee65a5-81e6-4736-91ac-6399ba34947e" (UID: "21ee65a5-81e6-4736-91ac-6399ba34947e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:36.051987 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:36.051930 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21ee65a5-81e6-4736-91ac-6399ba34947e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/21ee65a5-81e6-4736-91ac-6399ba34947e-dshm\") pod \"21ee65a5-81e6-4736-91ac-6399ba34947e\" (UID: \"21ee65a5-81e6-4736-91ac-6399ba34947e\") " Nov 02 01:45:36.052218 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:36.052165 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21ee65a5-81e6-4736-91ac-6399ba34947e-home" pod "21ee65a5-81e6-4736-91ac-6399ba34947e" (UID: "21ee65a5-81e6-4736-91ac-6399ba34947e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:36.052218 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:36.052179 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21ee65a5-81e6-4736-91ac-6399ba34947e-model-cache" pod "21ee65a5-81e6-4736-91ac-6399ba34947e" (UID: "21ee65a5-81e6-4736-91ac-6399ba34947e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:36.052935 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:36.052906 4427 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21ee65a5-81e6-4736-91ac-6399ba34947e-tmp-dir\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:36.053065 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:36.052940 4427 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21ee65a5-81e6-4736-91ac-6399ba34947e-model-cache\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:36.053065 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:36.052955 4427 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21ee65a5-81e6-4736-91ac-6399ba34947e-home\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:36.054623 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:36.054592 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21ee65a5-81e6-4736-91ac-6399ba34947e-dshm" pod "21ee65a5-81e6-4736-91ac-6399ba34947e" (UID: "21ee65a5-81e6-4736-91ac-6399ba34947e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:36.054767 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:36.054629 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21ee65a5-81e6-4736-91ac-6399ba34947e-kube-api-access-vzxpt" pod "21ee65a5-81e6-4736-91ac-6399ba34947e" (UID: "21ee65a5-81e6-4736-91ac-6399ba34947e"). InnerVolumeSpecName "kube-api-access-vzxpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:45:36.054767 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:36.054690 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21ee65a5-81e6-4736-91ac-6399ba34947e-tls-certs" pod "21ee65a5-81e6-4736-91ac-6399ba34947e" (UID: "21ee65a5-81e6-4736-91ac-6399ba34947e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:45:36.154291 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:36.154233 4427 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21ee65a5-81e6-4736-91ac-6399ba34947e-tls-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:36.154291 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:36.154292 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzxpt\" (UniqueName: \"kubernetes.io/projected/21ee65a5-81e6-4736-91ac-6399ba34947e-kube-api-access-vzxpt\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:36.154532 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:36.154312 4427 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21ee65a5-81e6-4736-91ac-6399ba34947e-dshm\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:36.605751 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:36.605719 4427 generic.go:356] "Generic (PLEG): container finished" podID="21ee65a5-81e6-4736-91ac-6399ba34947e" containerID="9630734ee18562779f539a58dd9f4825d78c0d80937338443fb9fe92cc889f11" exitCode=0 Nov 02 01:45:36.606242 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:36.605786 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-12e7cb/test-model-12e7cb-kserve-5b5d7468c9-xkkqs" Nov 02 01:45:36.606242 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:36.605812 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-12e7cb/test-model-12e7cb-kserve-5b5d7468c9-xkkqs" event={"ID":"21ee65a5-81e6-4736-91ac-6399ba34947e","Type":"ContainerDied","Data":"9630734ee18562779f539a58dd9f4825d78c0d80937338443fb9fe92cc889f11"} Nov 02 01:45:36.606242 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:36.605859 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-12e7cb/test-model-12e7cb-kserve-5b5d7468c9-xkkqs" event={"ID":"21ee65a5-81e6-4736-91ac-6399ba34947e","Type":"ContainerDied","Data":"07ae4c2090b707918781ca3f1dbb8174b215c40bbf6f32670ef099b0b98e2306"} Nov 02 01:45:36.606242 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:36.605877 4427 scope.go:122] "RemoveContainer" containerID="9630734ee18562779f539a58dd9f4825d78c0d80937338443fb9fe92cc889f11" Nov 02 01:45:36.620102 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:36.620080 4427 scope.go:122] "RemoveContainer" containerID="9630734ee18562779f539a58dd9f4825d78c0d80937338443fb9fe92cc889f11" Nov 02 01:45:36.620451 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:45:36.620431 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9630734ee18562779f539a58dd9f4825d78c0d80937338443fb9fe92cc889f11\": container with ID starting with 9630734ee18562779f539a58dd9f4825d78c0d80937338443fb9fe92cc889f11 not found: ID does not exist" containerID="9630734ee18562779f539a58dd9f4825d78c0d80937338443fb9fe92cc889f11" Nov 02 01:45:36.620537 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:36.620459 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9630734ee18562779f539a58dd9f4825d78c0d80937338443fb9fe92cc889f11"} err="failed to get container status \"9630734ee18562779f539a58dd9f4825d78c0d80937338443fb9fe92cc889f11\": rpc error: code = NotFound desc = could not find container \"9630734ee18562779f539a58dd9f4825d78c0d80937338443fb9fe92cc889f11\": container with ID starting with 9630734ee18562779f539a58dd9f4825d78c0d80937338443fb9fe92cc889f11 not found: ID does not exist" Nov 02 01:45:36.634179 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:36.634157 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-12e7cb/test-model-12e7cb-kserve-5b5d7468c9-xkkqs"] Nov 02 01:45:36.637502 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:36.637483 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-12e7cb/test-model-12e7cb-kserve-5b5d7468c9-xkkqs"] Nov 02 01:45:36.835710 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:36.835676 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht" Nov 02 01:45:36.838313 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:36.837169 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht" Nov 02 01:45:36.846008 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:36.845609 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht" Nov 02 01:45:37.534391 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:37.534343 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21ee65a5-81e6-4736-91ac-6399ba34947e" path="/var/lib/kubelet/pods/21ee65a5-81e6-4736-91ac-6399ba34947e/volumes" Nov 02 01:45:48.790072 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:48.790033 4427 generic.go:356] "Generic (PLEG): container finished" podID="1d6df883-72f0-4727-acbc-72bb9bd02f05" containerID="1998796f0577d5eb3cec5d2bcc5e6bcc70971c4401f42cf0951a7e59b1e3cc7c" exitCode=0 Nov 02 01:45:48.790547 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:48.790111 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp" event={"ID":"1d6df883-72f0-4727-acbc-72bb9bd02f05","Type":"ContainerDied","Data":"1998796f0577d5eb3cec5d2bcc5e6bcc70971c4401f42cf0951a7e59b1e3cc7c"} Nov 02 01:45:48.790670 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:48.790645 4427 scope.go:122] "RemoveContainer" containerID="1998796f0577d5eb3cec5d2bcc5e6bcc70971c4401f42cf0951a7e59b1e3cc7c" Nov 02 01:45:48.792329 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:48.792301 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:45:48.792440 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:48.792394 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:45:48.792440 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:48.792433 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:45:49.800703 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:49.800664 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp" event={"ID":"1d6df883-72f0-4727-acbc-72bb9bd02f05","Type":"ContainerStarted","Data":"b850b4e46248ca4510211fae40e97cf3e098413ce855ed741fc0742a4d94a80d"} Nov 02 01:45:50.332965 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:50.332922 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp" Nov 02 01:45:50.808465 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:50.804109 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp" Nov 02 01:45:50.814560 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:50.814056 4427 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 02 01:45:50.814560 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:50.814112 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp" podUID="1d6df883-72f0-4727-acbc-72bb9bd02f05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 02 01:45:50.821694 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:50.821663 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp"] Nov 02 01:45:50.823581 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:50.823520 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp" podUID="c53652cf-999e-4b99-b2f7-e779912799ce" containerName="istio-proxy" containerID="cri-o://94b6875f8e746e059d65e1b3ea6995d7a8bf6fc3fa03b93c44d13259dcd7901b" gracePeriod=30 Nov 02 01:45:50.827875 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:50.827843 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp"] Nov 02 01:45:50.828256 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:50.828182 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp" podUID="907ed6a0-8a68-49c1-aa49-916b377a90f5" containerName="istio-proxy" containerID="cri-o://7cc0cf8471ef328018af21be9b9a9a2880980c333b2561ea4ba40ee7f4ff572c" gracePeriod=30 Nov 02 01:45:50.834063 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:50.834031 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm"] Nov 02 01:45:50.834367 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:50.834327 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm" podUID="47e0dc4b-619a-4759-95af-c288b7257966" containerName="istio-proxy" containerID="cri-o://f8c6654ffb98ea4839b23f6da6887cd9200bdc6919b65dd81d2f46d364d9c679" gracePeriod=30 Nov 02 01:45:51.779401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:51.777830 4427 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 02 01:45:51.779401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:51.777913 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp" podUID="1d6df883-72f0-4727-acbc-72bb9bd02f05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 02 01:45:52.771401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:52.769750 4427 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 02 01:45:52.771401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:52.769833 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp" podUID="1d6df883-72f0-4727-acbc-72bb9bd02f05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 02 01:45:53.770844 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:53.770805 4427 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 02 01:45:53.771174 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:53.771139 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp" podUID="1d6df883-72f0-4727-acbc-72bb9bd02f05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 02 01:45:54.635364 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:54.635324 4427 patch_prober.go:29] interesting pod/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 01:45:54.635800 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:54.635413 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp" podUID="907ed6a0-8a68-49c1-aa49-916b377a90f5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 01:45:54.774874 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:54.774828 4427 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 02 01:45:54.775094 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:54.774901 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp" podUID="1d6df883-72f0-4727-acbc-72bb9bd02f05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 02 01:45:54.958033 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:54.957939 4427 patch_prober.go:29] interesting pod/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 01:45:54.958033 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:54.958007 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm" podUID="47e0dc4b-619a-4759-95af-c288b7257966" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 01:45:55.774648 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:55.774616 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp" Nov 02 01:45:55.776050 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:55.776027 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp" Nov 02 01:45:56.194329 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.194297 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp" Nov 02 01:45:56.199099 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.199070 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp" Nov 02 01:45:56.269741 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.269717 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm" Nov 02 01:45:56.279147 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.279119 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/907ed6a0-8a68-49c1-aa49-916b377a90f5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/907ed6a0-8a68-49c1-aa49-916b377a90f5-istiod-ca-cert\") pod \"907ed6a0-8a68-49c1-aa49-916b377a90f5\" (UID: \"907ed6a0-8a68-49c1-aa49-916b377a90f5\") " Nov 02 01:45:56.279310 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.279157 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/907ed6a0-8a68-49c1-aa49-916b377a90f5-kube-api-access-cvnq9\" (UniqueName: \"kubernetes.io/projected/907ed6a0-8a68-49c1-aa49-916b377a90f5-kube-api-access-cvnq9\") pod \"907ed6a0-8a68-49c1-aa49-916b377a90f5\" (UID: \"907ed6a0-8a68-49c1-aa49-916b377a90f5\") " Nov 02 01:45:56.279310 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.279182 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/907ed6a0-8a68-49c1-aa49-916b377a90f5-istio-token\" (UniqueName: \"kubernetes.io/projected/907ed6a0-8a68-49c1-aa49-916b377a90f5-istio-token\") pod \"907ed6a0-8a68-49c1-aa49-916b377a90f5\" (UID: \"907ed6a0-8a68-49c1-aa49-916b377a90f5\") " Nov 02 01:45:56.279310 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.279206 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/907ed6a0-8a68-49c1-aa49-916b377a90f5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/907ed6a0-8a68-49c1-aa49-916b377a90f5-credential-socket\") pod \"907ed6a0-8a68-49c1-aa49-916b377a90f5\" (UID: \"907ed6a0-8a68-49c1-aa49-916b377a90f5\") " Nov 02 01:45:56.279310 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.279287 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/907ed6a0-8a68-49c1-aa49-916b377a90f5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/907ed6a0-8a68-49c1-aa49-916b377a90f5-istio-envoy\") pod \"907ed6a0-8a68-49c1-aa49-916b377a90f5\" (UID: \"907ed6a0-8a68-49c1-aa49-916b377a90f5\") " Nov 02 01:45:56.279607 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.279444 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c53652cf-999e-4b99-b2f7-e779912799ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c53652cf-999e-4b99-b2f7-e779912799ce-workload-socket\") pod \"c53652cf-999e-4b99-b2f7-e779912799ce\" (UID: \"c53652cf-999e-4b99-b2f7-e779912799ce\") " Nov 02 01:45:56.279607 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.279470 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c53652cf-999e-4b99-b2f7-e779912799ce-istio-token\" (UniqueName: \"kubernetes.io/projected/c53652cf-999e-4b99-b2f7-e779912799ce-istio-token\") pod \"c53652cf-999e-4b99-b2f7-e779912799ce\" (UID: \"c53652cf-999e-4b99-b2f7-e779912799ce\") " Nov 02 01:45:56.279607 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.279521 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c53652cf-999e-4b99-b2f7-e779912799ce-kube-api-access-ptclz\" (UniqueName: \"kubernetes.io/projected/c53652cf-999e-4b99-b2f7-e779912799ce-kube-api-access-ptclz\") pod \"c53652cf-999e-4b99-b2f7-e779912799ce\" (UID: \"c53652cf-999e-4b99-b2f7-e779912799ce\") " Nov 02 01:45:56.279607 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.279547 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c53652cf-999e-4b99-b2f7-e779912799ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c53652cf-999e-4b99-b2f7-e779912799ce-istiod-ca-cert\") pod \"c53652cf-999e-4b99-b2f7-e779912799ce\" (UID: \"c53652cf-999e-4b99-b2f7-e779912799ce\") " Nov 02 01:45:56.279607 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.279584 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/907ed6a0-8a68-49c1-aa49-916b377a90f5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/907ed6a0-8a68-49c1-aa49-916b377a90f5-istio-data\") pod \"907ed6a0-8a68-49c1-aa49-916b377a90f5\" (UID: \"907ed6a0-8a68-49c1-aa49-916b377a90f5\") " Nov 02 01:45:56.279876 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.279618 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/907ed6a0-8a68-49c1-aa49-916b377a90f5-istiod-ca-cert" pod "907ed6a0-8a68-49c1-aa49-916b377a90f5" (UID: "907ed6a0-8a68-49c1-aa49-916b377a90f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:45:56.279876 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.279614 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/907ed6a0-8a68-49c1-aa49-916b377a90f5-credential-socket" pod "907ed6a0-8a68-49c1-aa49-916b377a90f5" (UID: "907ed6a0-8a68-49c1-aa49-916b377a90f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:56.279876 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.279820 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/907ed6a0-8a68-49c1-aa49-916b377a90f5-istio-data" pod "907ed6a0-8a68-49c1-aa49-916b377a90f5" (UID: "907ed6a0-8a68-49c1-aa49-916b377a90f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:56.279876 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.279864 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c53652cf-999e-4b99-b2f7-e779912799ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c53652cf-999e-4b99-b2f7-e779912799ce-istio-data\") pod \"c53652cf-999e-4b99-b2f7-e779912799ce\" (UID: \"c53652cf-999e-4b99-b2f7-e779912799ce\") " Nov 02 01:45:56.280057 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.280032 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c53652cf-999e-4b99-b2f7-e779912799ce-istiod-ca-cert" pod "c53652cf-999e-4b99-b2f7-e779912799ce" (UID: "c53652cf-999e-4b99-b2f7-e779912799ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:45:56.280124 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.280103 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c53652cf-999e-4b99-b2f7-e779912799ce-workload-socket" pod "c53652cf-999e-4b99-b2f7-e779912799ce" (UID: "c53652cf-999e-4b99-b2f7-e779912799ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:56.280176 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.280125 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c53652cf-999e-4b99-b2f7-e779912799ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c53652cf-999e-4b99-b2f7-e779912799ce-istio-envoy\") pod \"c53652cf-999e-4b99-b2f7-e779912799ce\" (UID: \"c53652cf-999e-4b99-b2f7-e779912799ce\") " Nov 02 01:45:56.280176 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.280160 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/907ed6a0-8a68-49c1-aa49-916b377a90f5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/907ed6a0-8a68-49c1-aa49-916b377a90f5-istio-podinfo\") pod \"907ed6a0-8a68-49c1-aa49-916b377a90f5\" (UID: \"907ed6a0-8a68-49c1-aa49-916b377a90f5\") " Nov 02 01:45:56.280472 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.280195 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c53652cf-999e-4b99-b2f7-e779912799ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c53652cf-999e-4b99-b2f7-e779912799ce-credential-socket\") pod \"c53652cf-999e-4b99-b2f7-e779912799ce\" (UID: \"c53652cf-999e-4b99-b2f7-e779912799ce\") " Nov 02 01:45:56.280472 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.280227 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c53652cf-999e-4b99-b2f7-e779912799ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c53652cf-999e-4b99-b2f7-e779912799ce-istio-podinfo\") pod \"c53652cf-999e-4b99-b2f7-e779912799ce\" (UID: \"c53652cf-999e-4b99-b2f7-e779912799ce\") " Nov 02 01:45:56.280472 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.280409 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c53652cf-999e-4b99-b2f7-e779912799ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c53652cf-999e-4b99-b2f7-e779912799ce-workload-certs\") pod \"c53652cf-999e-4b99-b2f7-e779912799ce\" (UID: \"c53652cf-999e-4b99-b2f7-e779912799ce\") " Nov 02 01:45:56.280472 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.280468 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/907ed6a0-8a68-49c1-aa49-916b377a90f5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/907ed6a0-8a68-49c1-aa49-916b377a90f5-workload-certs\") pod \"907ed6a0-8a68-49c1-aa49-916b377a90f5\" (UID: \"907ed6a0-8a68-49c1-aa49-916b377a90f5\") " Nov 02 01:45:56.280788 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.280504 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/907ed6a0-8a68-49c1-aa49-916b377a90f5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/907ed6a0-8a68-49c1-aa49-916b377a90f5-workload-socket\") pod \"907ed6a0-8a68-49c1-aa49-916b377a90f5\" (UID: \"907ed6a0-8a68-49c1-aa49-916b377a90f5\") " Nov 02 01:45:56.280788 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.280566 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c53652cf-999e-4b99-b2f7-e779912799ce-istio-data" pod "c53652cf-999e-4b99-b2f7-e779912799ce" (UID: "c53652cf-999e-4b99-b2f7-e779912799ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:56.281168 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.281138 4427 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/907ed6a0-8a68-49c1-aa49-916b377a90f5-istiod-ca-cert\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:56.281266 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.281172 4427 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/907ed6a0-8a68-49c1-aa49-916b377a90f5-credential-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:56.281266 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.281191 4427 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c53652cf-999e-4b99-b2f7-e779912799ce-workload-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:56.281266 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.281206 4427 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c53652cf-999e-4b99-b2f7-e779912799ce-istiod-ca-cert\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:56.281266 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.281220 4427 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/907ed6a0-8a68-49c1-aa49-916b377a90f5-istio-data\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:56.281266 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.281235 4427 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c53652cf-999e-4b99-b2f7-e779912799ce-istio-data\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:56.281564 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.281457 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/907ed6a0-8a68-49c1-aa49-916b377a90f5-workload-socket" pod "907ed6a0-8a68-49c1-aa49-916b377a90f5" (UID: "907ed6a0-8a68-49c1-aa49-916b377a90f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:56.281671 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.281602 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c53652cf-999e-4b99-b2f7-e779912799ce-workload-certs" pod "c53652cf-999e-4b99-b2f7-e779912799ce" (UID: "c53652cf-999e-4b99-b2f7-e779912799ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:56.281742 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.281665 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c53652cf-999e-4b99-b2f7-e779912799ce-credential-socket" pod "c53652cf-999e-4b99-b2f7-e779912799ce" (UID: "c53652cf-999e-4b99-b2f7-e779912799ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:56.281742 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.281682 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/907ed6a0-8a68-49c1-aa49-916b377a90f5-workload-certs" pod "907ed6a0-8a68-49c1-aa49-916b377a90f5" (UID: "907ed6a0-8a68-49c1-aa49-916b377a90f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:56.282847 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.282818 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c53652cf-999e-4b99-b2f7-e779912799ce-kube-api-access-ptclz" pod "c53652cf-999e-4b99-b2f7-e779912799ce" (UID: "c53652cf-999e-4b99-b2f7-e779912799ce"). InnerVolumeSpecName "kube-api-access-ptclz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:45:56.283578 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.283554 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/907ed6a0-8a68-49c1-aa49-916b377a90f5-istio-envoy" pod "907ed6a0-8a68-49c1-aa49-916b377a90f5" (UID: "907ed6a0-8a68-49c1-aa49-916b377a90f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:56.284124 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.284088 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/907ed6a0-8a68-49c1-aa49-916b377a90f5-istio-token" pod "907ed6a0-8a68-49c1-aa49-916b377a90f5" (UID: "907ed6a0-8a68-49c1-aa49-916b377a90f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:45:56.284124 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.284103 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c53652cf-999e-4b99-b2f7-e779912799ce-istio-podinfo" pod "c53652cf-999e-4b99-b2f7-e779912799ce" (UID: "c53652cf-999e-4b99-b2f7-e779912799ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:45:56.284297 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.284133 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c53652cf-999e-4b99-b2f7-e779912799ce-istio-token" pod "c53652cf-999e-4b99-b2f7-e779912799ce" (UID: "c53652cf-999e-4b99-b2f7-e779912799ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:45:56.284532 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.284506 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/907ed6a0-8a68-49c1-aa49-916b377a90f5-kube-api-access-cvnq9" pod "907ed6a0-8a68-49c1-aa49-916b377a90f5" (UID: "907ed6a0-8a68-49c1-aa49-916b377a90f5"). InnerVolumeSpecName "kube-api-access-cvnq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:45:56.284628 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.284611 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c53652cf-999e-4b99-b2f7-e779912799ce-istio-envoy" pod "c53652cf-999e-4b99-b2f7-e779912799ce" (UID: "c53652cf-999e-4b99-b2f7-e779912799ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:56.285430 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.285404 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/907ed6a0-8a68-49c1-aa49-916b377a90f5-istio-podinfo" pod "907ed6a0-8a68-49c1-aa49-916b377a90f5" (UID: "907ed6a0-8a68-49c1-aa49-916b377a90f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:45:56.382683 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.382638 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47e0dc4b-619a-4759-95af-c288b7257966-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47e0dc4b-619a-4759-95af-c288b7257966-istio-envoy\") pod \"47e0dc4b-619a-4759-95af-c288b7257966\" (UID: \"47e0dc4b-619a-4759-95af-c288b7257966\") " Nov 02 01:45:56.382862 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.382845 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47e0dc4b-619a-4759-95af-c288b7257966-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47e0dc4b-619a-4759-95af-c288b7257966-istio-data\") pod \"47e0dc4b-619a-4759-95af-c288b7257966\" (UID: \"47e0dc4b-619a-4759-95af-c288b7257966\") " Nov 02 01:45:56.382934 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.382917 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47e0dc4b-619a-4759-95af-c288b7257966-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e0dc4b-619a-4759-95af-c288b7257966-workload-socket\") pod \"47e0dc4b-619a-4759-95af-c288b7257966\" (UID: \"47e0dc4b-619a-4759-95af-c288b7257966\") " Nov 02 01:45:56.383007 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.382959 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47e0dc4b-619a-4759-95af-c288b7257966-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e0dc4b-619a-4759-95af-c288b7257966-credential-socket\") pod \"47e0dc4b-619a-4759-95af-c288b7257966\" (UID: \"47e0dc4b-619a-4759-95af-c288b7257966\") " Nov 02 01:45:56.383100 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.383075 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47e0dc4b-619a-4759-95af-c288b7257966-kube-api-access-scml5\" (UniqueName: \"kubernetes.io/projected/47e0dc4b-619a-4759-95af-c288b7257966-kube-api-access-scml5\") pod \"47e0dc4b-619a-4759-95af-c288b7257966\" (UID: \"47e0dc4b-619a-4759-95af-c288b7257966\") " Nov 02 01:45:56.383188 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.383122 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47e0dc4b-619a-4759-95af-c288b7257966-istio-token\" (UniqueName: \"kubernetes.io/projected/47e0dc4b-619a-4759-95af-c288b7257966-istio-token\") pod \"47e0dc4b-619a-4759-95af-c288b7257966\" (UID: \"47e0dc4b-619a-4759-95af-c288b7257966\") " Nov 02 01:45:56.383188 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.383175 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/47e0dc4b-619a-4759-95af-c288b7257966-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47e0dc4b-619a-4759-95af-c288b7257966-istio-podinfo\") pod \"47e0dc4b-619a-4759-95af-c288b7257966\" (UID: \"47e0dc4b-619a-4759-95af-c288b7257966\") " Nov 02 01:45:56.383294 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.383206 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47e0dc4b-619a-4759-95af-c288b7257966-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47e0dc4b-619a-4759-95af-c288b7257966-istiod-ca-cert\") pod \"47e0dc4b-619a-4759-95af-c288b7257966\" (UID: \"47e0dc4b-619a-4759-95af-c288b7257966\") " Nov 02 01:45:56.383294 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.383260 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47e0dc4b-619a-4759-95af-c288b7257966-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47e0dc4b-619a-4759-95af-c288b7257966-workload-certs\") pod \"47e0dc4b-619a-4759-95af-c288b7257966\" (UID: \"47e0dc4b-619a-4759-95af-c288b7257966\") " Nov 02 01:45:56.383294 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.383259 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e0dc4b-619a-4759-95af-c288b7257966-workload-socket" pod "47e0dc4b-619a-4759-95af-c288b7257966" (UID: "47e0dc4b-619a-4759-95af-c288b7257966"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:56.383496 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.383297 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e0dc4b-619a-4759-95af-c288b7257966-credential-socket" pod "47e0dc4b-619a-4759-95af-c288b7257966" (UID: "47e0dc4b-619a-4759-95af-c288b7257966"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:56.383496 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.383169 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e0dc4b-619a-4759-95af-c288b7257966-istio-data" pod "47e0dc4b-619a-4759-95af-c288b7257966" (UID: "47e0dc4b-619a-4759-95af-c288b7257966"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:56.383851 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.383828 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47e0dc4b-619a-4759-95af-c288b7257966-istiod-ca-cert" pod "47e0dc4b-619a-4759-95af-c288b7257966" (UID: "47e0dc4b-619a-4759-95af-c288b7257966"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:45:56.383851 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.383838 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e0dc4b-619a-4759-95af-c288b7257966-workload-certs" pod "47e0dc4b-619a-4759-95af-c288b7257966" (UID: "47e0dc4b-619a-4759-95af-c288b7257966"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:56.384042 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.383858 4427 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/907ed6a0-8a68-49c1-aa49-916b377a90f5-istio-envoy\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:56.384042 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.383884 4427 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c53652cf-999e-4b99-b2f7-e779912799ce-istio-token\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:56.384042 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.383901 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptclz\" (UniqueName: \"kubernetes.io/projected/c53652cf-999e-4b99-b2f7-e779912799ce-kube-api-access-ptclz\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:56.384042 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.383917 4427 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47e0dc4b-619a-4759-95af-c288b7257966-istio-data\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:56.384042 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.383932 4427 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c53652cf-999e-4b99-b2f7-e779912799ce-istio-envoy\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:56.384042 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.383947 4427 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/907ed6a0-8a68-49c1-aa49-916b377a90f5-istio-podinfo\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:56.384042 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.383963 4427 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c53652cf-999e-4b99-b2f7-e779912799ce-credential-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:56.384042 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.383978 4427 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c53652cf-999e-4b99-b2f7-e779912799ce-istio-podinfo\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:56.384042 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.383990 4427 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e0dc4b-619a-4759-95af-c288b7257966-workload-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:56.384042 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.384005 4427 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e0dc4b-619a-4759-95af-c288b7257966-credential-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:56.384042 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.384018 4427 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c53652cf-999e-4b99-b2f7-e779912799ce-workload-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:56.384042 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.384031 4427 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/907ed6a0-8a68-49c1-aa49-916b377a90f5-workload-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:56.384042 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.384045 4427 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/907ed6a0-8a68-49c1-aa49-916b377a90f5-workload-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:56.385061 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.384061 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvnq9\" (UniqueName: \"kubernetes.io/projected/907ed6a0-8a68-49c1-aa49-916b377a90f5-kube-api-access-cvnq9\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:56.385061 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.384076 4427 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/907ed6a0-8a68-49c1-aa49-916b377a90f5-istio-token\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:56.385888 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.385856 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47e0dc4b-619a-4759-95af-c288b7257966-istio-token" pod "47e0dc4b-619a-4759-95af-c288b7257966" (UID: "47e0dc4b-619a-4759-95af-c288b7257966"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:45:56.385989 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.385934 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47e0dc4b-619a-4759-95af-c288b7257966-istio-podinfo" pod "47e0dc4b-619a-4759-95af-c288b7257966" (UID: "47e0dc4b-619a-4759-95af-c288b7257966"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:45:56.386148 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.386130 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47e0dc4b-619a-4759-95af-c288b7257966-kube-api-access-scml5" pod "47e0dc4b-619a-4759-95af-c288b7257966" (UID: "47e0dc4b-619a-4759-95af-c288b7257966"). InnerVolumeSpecName "kube-api-access-scml5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:45:56.386281 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.386258 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e0dc4b-619a-4759-95af-c288b7257966-istio-envoy" pod "47e0dc4b-619a-4759-95af-c288b7257966" (UID: "47e0dc4b-619a-4759-95af-c288b7257966"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:45:56.485387 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.485350 4427 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47e0dc4b-619a-4759-95af-c288b7257966-istio-envoy\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:56.485562 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.485396 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scml5\" (UniqueName: \"kubernetes.io/projected/47e0dc4b-619a-4759-95af-c288b7257966-kube-api-access-scml5\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:56.485562 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.485408 4427 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47e0dc4b-619a-4759-95af-c288b7257966-istio-token\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:56.485562 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.485417 4427 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47e0dc4b-619a-4759-95af-c288b7257966-istio-podinfo\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:56.485562 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.485428 4427 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47e0dc4b-619a-4759-95af-c288b7257966-istiod-ca-cert\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:56.485562 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.485436 4427 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47e0dc4b-619a-4759-95af-c288b7257966-workload-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:45:56.880126 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.880083 4427 generic.go:356] "Generic (PLEG): container finished" podID="c53652cf-999e-4b99-b2f7-e779912799ce" containerID="94b6875f8e746e059d65e1b3ea6995d7a8bf6fc3fa03b93c44d13259dcd7901b" exitCode=0 Nov 02 01:45:56.880601 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.880150 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp" event={"ID":"c53652cf-999e-4b99-b2f7-e779912799ce","Type":"ContainerDied","Data":"94b6875f8e746e059d65e1b3ea6995d7a8bf6fc3fa03b93c44d13259dcd7901b"} Nov 02 01:45:56.880601 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.880221 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp" event={"ID":"c53652cf-999e-4b99-b2f7-e779912799ce","Type":"ContainerDied","Data":"6433ce90e1b55a9ea7157c84b47a4f04d69e519b08a2de6e25def154048752a5"} Nov 02 01:45:56.880601 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.880157 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp" Nov 02 01:45:56.880601 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.880250 4427 scope.go:122] "RemoveContainer" containerID="94b6875f8e746e059d65e1b3ea6995d7a8bf6fc3fa03b93c44d13259dcd7901b" Nov 02 01:45:56.882087 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.882063 4427 generic.go:356] "Generic (PLEG): container finished" podID="907ed6a0-8a68-49c1-aa49-916b377a90f5" containerID="7cc0cf8471ef328018af21be9b9a9a2880980c333b2561ea4ba40ee7f4ff572c" exitCode=0 Nov 02 01:45:56.882221 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.882113 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp" Nov 02 01:45:56.882221 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.882156 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp" event={"ID":"907ed6a0-8a68-49c1-aa49-916b377a90f5","Type":"ContainerDied","Data":"7cc0cf8471ef328018af21be9b9a9a2880980c333b2561ea4ba40ee7f4ff572c"} Nov 02 01:45:56.882221 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.882213 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp" event={"ID":"907ed6a0-8a68-49c1-aa49-916b377a90f5","Type":"ContainerDied","Data":"56069eab0114b611733a513bd1f9ba70a98a439a8289fdf91231da5784cc178c"} Nov 02 01:45:56.885188 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.885105 4427 generic.go:356] "Generic (PLEG): container finished" podID="47e0dc4b-619a-4759-95af-c288b7257966" containerID="f8c6654ffb98ea4839b23f6da6887cd9200bdc6919b65dd81d2f46d364d9c679" exitCode=0 Nov 02 01:45:56.885188 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.885152 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm" Nov 02 01:45:56.885329 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.885194 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm" event={"ID":"47e0dc4b-619a-4759-95af-c288b7257966","Type":"ContainerDied","Data":"f8c6654ffb98ea4839b23f6da6887cd9200bdc6919b65dd81d2f46d364d9c679"} Nov 02 01:45:56.885329 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.885234 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm" event={"ID":"47e0dc4b-619a-4759-95af-c288b7257966","Type":"ContainerDied","Data":"e16a78e89c63f614d898d72ee801ae0caeb2b4bfa2161402814b7a83d9208d0a"} Nov 02 01:45:56.893870 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.893845 4427 scope.go:122] "RemoveContainer" containerID="94b6875f8e746e059d65e1b3ea6995d7a8bf6fc3fa03b93c44d13259dcd7901b" Nov 02 01:45:56.894310 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:45:56.894216 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94b6875f8e746e059d65e1b3ea6995d7a8bf6fc3fa03b93c44d13259dcd7901b\": container with ID starting with 94b6875f8e746e059d65e1b3ea6995d7a8bf6fc3fa03b93c44d13259dcd7901b not found: ID does not exist" containerID="94b6875f8e746e059d65e1b3ea6995d7a8bf6fc3fa03b93c44d13259dcd7901b" Nov 02 01:45:56.894310 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.894250 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94b6875f8e746e059d65e1b3ea6995d7a8bf6fc3fa03b93c44d13259dcd7901b"} err="failed to get container status \"94b6875f8e746e059d65e1b3ea6995d7a8bf6fc3fa03b93c44d13259dcd7901b\": rpc error: code = NotFound desc = could not find container \"94b6875f8e746e059d65e1b3ea6995d7a8bf6fc3fa03b93c44d13259dcd7901b\": container with ID starting with 94b6875f8e746e059d65e1b3ea6995d7a8bf6fc3fa03b93c44d13259dcd7901b not found: ID does not exist" Nov 02 01:45:56.894310 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.894273 4427 scope.go:122] "RemoveContainer" containerID="7cc0cf8471ef328018af21be9b9a9a2880980c333b2561ea4ba40ee7f4ff572c" Nov 02 01:45:56.906007 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.905983 4427 scope.go:122] "RemoveContainer" containerID="7cc0cf8471ef328018af21be9b9a9a2880980c333b2561ea4ba40ee7f4ff572c" Nov 02 01:45:56.906306 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:45:56.906289 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cc0cf8471ef328018af21be9b9a9a2880980c333b2561ea4ba40ee7f4ff572c\": container with ID starting with 7cc0cf8471ef328018af21be9b9a9a2880980c333b2561ea4ba40ee7f4ff572c not found: ID does not exist" containerID="7cc0cf8471ef328018af21be9b9a9a2880980c333b2561ea4ba40ee7f4ff572c" Nov 02 01:45:56.906437 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.906314 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cc0cf8471ef328018af21be9b9a9a2880980c333b2561ea4ba40ee7f4ff572c"} err="failed to get container status \"7cc0cf8471ef328018af21be9b9a9a2880980c333b2561ea4ba40ee7f4ff572c\": rpc error: code = NotFound desc = could not find container \"7cc0cf8471ef328018af21be9b9a9a2880980c333b2561ea4ba40ee7f4ff572c\": container with ID starting with 7cc0cf8471ef328018af21be9b9a9a2880980c333b2561ea4ba40ee7f4ff572c not found: ID does not exist" Nov 02 01:45:56.906437 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.906333 4427 scope.go:122] "RemoveContainer" containerID="f8c6654ffb98ea4839b23f6da6887cd9200bdc6919b65dd81d2f46d364d9c679" Nov 02 01:45:56.913628 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.913597 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp"] Nov 02 01:45:56.917407 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.917349 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-fkbzp"] Nov 02 01:45:56.918810 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.918790 4427 scope.go:122] "RemoveContainer" containerID="f8c6654ffb98ea4839b23f6da6887cd9200bdc6919b65dd81d2f46d364d9c679" Nov 02 01:45:56.919083 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:45:56.919065 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8c6654ffb98ea4839b23f6da6887cd9200bdc6919b65dd81d2f46d364d9c679\": container with ID starting with f8c6654ffb98ea4839b23f6da6887cd9200bdc6919b65dd81d2f46d364d9c679 not found: ID does not exist" containerID="f8c6654ffb98ea4839b23f6da6887cd9200bdc6919b65dd81d2f46d364d9c679" Nov 02 01:45:56.919136 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.919089 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8c6654ffb98ea4839b23f6da6887cd9200bdc6919b65dd81d2f46d364d9c679"} err="failed to get container status \"f8c6654ffb98ea4839b23f6da6887cd9200bdc6919b65dd81d2f46d364d9c679\": rpc error: code = NotFound desc = could not find container \"f8c6654ffb98ea4839b23f6da6887cd9200bdc6919b65dd81d2f46d364d9c679\": container with ID starting with f8c6654ffb98ea4839b23f6da6887cd9200bdc6919b65dd81d2f46d364d9c679 not found: ID does not exist" Nov 02 01:45:56.926578 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.926560 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp"] Nov 02 01:45:56.931278 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.931255 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-x22mp"] Nov 02 01:45:56.943018 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.942942 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm"] Nov 02 01:45:56.946932 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:56.946909 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-bf1e55-openshift-default-75c76ccf59-bjmlm"] Nov 02 01:45:57.534626 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:57.534587 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47e0dc4b-619a-4759-95af-c288b7257966" path="/var/lib/kubelet/pods/47e0dc4b-619a-4759-95af-c288b7257966/volumes" Nov 02 01:45:57.535144 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:57.535122 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="907ed6a0-8a68-49c1-aa49-916b377a90f5" path="/var/lib/kubelet/pods/907ed6a0-8a68-49c1-aa49-916b377a90f5/volumes" Nov 02 01:45:57.535637 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:57.535618 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c53652cf-999e-4b99-b2f7-e779912799ce" path="/var/lib/kubelet/pods/c53652cf-999e-4b99-b2f7-e779912799ce/volumes" Nov 02 01:45:59.749394 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.744494 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22"] Nov 02 01:45:59.749394 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.745417 4427 cpu_manager.go:414] "containerMap: removing container" podUID="5779a226-eaa5-4428-99d1-47e4936591a2" containerName="cleanup" Nov 02 01:45:59.749394 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.745440 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5779a226-eaa5-4428-99d1-47e4936591a2" containerName="cleanup" Nov 02 01:45:59.749394 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.745465 4427 cpu_manager.go:414] "containerMap: removing container" podUID="82d82a81-f1df-446d-b07b-20f510354a7e" containerName="istio-proxy" Nov 02 01:45:59.749394 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.745473 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82d82a81-f1df-446d-b07b-20f510354a7e" containerName="istio-proxy" Nov 02 01:45:59.749394 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.745482 4427 cpu_manager.go:414] "containerMap: removing container" podUID="bcd48699-8090-4574-934c-b29668788243" containerName="cleanup" Nov 02 01:45:59.749394 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.745490 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcd48699-8090-4574-934c-b29668788243" containerName="cleanup" Nov 02 01:45:59.749394 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.745517 4427 cpu_manager.go:414] "containerMap: removing container" podUID="47e0dc4b-619a-4759-95af-c288b7257966" containerName="istio-proxy" Nov 02 01:45:59.749394 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.745525 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47e0dc4b-619a-4759-95af-c288b7257966" containerName="istio-proxy" Nov 02 01:45:59.749394 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.745539 4427 cpu_manager.go:414] "containerMap: removing container" podUID="886fa3e6-9143-4823-903d-7e68d06100ee" containerName="cleanup" Nov 02 01:45:59.749394 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.745546 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="886fa3e6-9143-4823-903d-7e68d06100ee" containerName="cleanup" Nov 02 01:45:59.749394 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.745571 4427 cpu_manager.go:414] "containerMap: removing container" podUID="d9f64fd6-8cb1-4a34-99f0-828c753a1119" containerName="istio-proxy" Nov 02 01:45:59.749394 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.745578 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9f64fd6-8cb1-4a34-99f0-828c753a1119" containerName="istio-proxy" Nov 02 01:45:59.749394 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.745601 4427 cpu_manager.go:414] "containerMap: removing container" podUID="36cac416-8352-4653-8ec9-7ad64348afa7" containerName="istio-proxy" Nov 02 01:45:59.749394 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.745608 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36cac416-8352-4653-8ec9-7ad64348afa7" containerName="istio-proxy" Nov 02 01:45:59.749394 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.745620 4427 cpu_manager.go:414] "containerMap: removing container" podUID="907ed6a0-8a68-49c1-aa49-916b377a90f5" containerName="istio-proxy" Nov 02 01:45:59.749394 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.745627 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="907ed6a0-8a68-49c1-aa49-916b377a90f5" containerName="istio-proxy" Nov 02 01:45:59.749394 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.745642 4427 cpu_manager.go:414] "containerMap: removing container" podUID="21ee65a5-81e6-4736-91ac-6399ba34947e" containerName="main" Nov 02 01:45:59.749394 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.745649 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21ee65a5-81e6-4736-91ac-6399ba34947e" containerName="main" Nov 02 01:45:59.749394 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.745663 4427 cpu_manager.go:414] "containerMap: removing container" podUID="c53652cf-999e-4b99-b2f7-e779912799ce" containerName="istio-proxy" Nov 02 01:45:59.749394 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.745670 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c53652cf-999e-4b99-b2f7-e779912799ce" containerName="istio-proxy" Nov 02 01:45:59.749394 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.745691 4427 cpu_manager.go:414] "containerMap: removing container" podUID="136eabea-7a63-4d89-bcc1-0e3d6c8810cf" containerName="istio-proxy" Nov 02 01:45:59.749394 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.745698 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="136eabea-7a63-4d89-bcc1-0e3d6c8810cf" containerName="istio-proxy" Nov 02 01:45:59.749394 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.745852 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="36cac416-8352-4653-8ec9-7ad64348afa7" containerName="istio-proxy" Nov 02 01:45:59.749394 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.745865 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="907ed6a0-8a68-49c1-aa49-916b377a90f5" containerName="istio-proxy" Nov 02 01:45:59.749394 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.745876 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="21ee65a5-81e6-4736-91ac-6399ba34947e" containerName="main" Nov 02 01:45:59.749394 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.745885 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="c53652cf-999e-4b99-b2f7-e779912799ce" containerName="istio-proxy" Nov 02 01:45:59.749394 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.745897 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="635a1467-2a08-4b8b-a8c0-b16e2a361888" containerName="cleanup" Nov 02 01:45:59.749394 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.745906 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="136eabea-7a63-4d89-bcc1-0e3d6c8810cf" containerName="istio-proxy" Nov 02 01:45:59.749394 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.745919 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="82d82a81-f1df-446d-b07b-20f510354a7e" containerName="istio-proxy" Nov 02 01:45:59.749394 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.745929 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="d983a998-461c-4341-aea9-1eb65375bbb7" containerName="cleanup" Nov 02 01:45:59.749394 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.745937 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="47e0dc4b-619a-4759-95af-c288b7257966" containerName="istio-proxy" Nov 02 01:45:59.749394 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.745946 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="5779a226-eaa5-4428-99d1-47e4936591a2" containerName="cleanup" Nov 02 01:45:59.749394 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.745955 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcd48699-8090-4574-934c-b29668788243" containerName="cleanup" Nov 02 01:45:59.749394 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.745962 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9f64fd6-8cb1-4a34-99f0-828c753a1119" containerName="istio-proxy" Nov 02 01:45:59.749394 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.745972 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="886fa3e6-9143-4823-903d-7e68d06100ee" containerName="cleanup" Nov 02 01:45:59.749394 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.745980 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="747eccfe-eecb-4cec-99a9-aea7857e1a34" containerName="cleanup" Nov 02 01:45:59.784867 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.783945 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22"] Nov 02 01:45:59.784867 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.784080 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22" Nov 02 01:45:59.791404 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.788711 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-eaec8c-openshift-default-dockercfg-scjs9\"" Nov 02 01:45:59.858809 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.858770 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f31dc5a1-ea54-4858-ac4c-2b990192ee4c-istio-envoy\") pod \"e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22\" (UID: \"f31dc5a1-ea54-4858-ac4c-2b990192ee4c\") " pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22" Nov 02 01:45:59.859016 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.858821 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f31dc5a1-ea54-4858-ac4c-2b990192ee4c-istio-podinfo\") pod \"e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22\" (UID: \"f31dc5a1-ea54-4858-ac4c-2b990192ee4c\") " pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22" Nov 02 01:45:59.859016 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.858941 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f31dc5a1-ea54-4858-ac4c-2b990192ee4c-workload-certs\") pod \"e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22\" (UID: \"f31dc5a1-ea54-4858-ac4c-2b990192ee4c\") " pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22" Nov 02 01:45:59.859016 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.858961 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns6dn\" (UniqueName: \"kubernetes.io/projected/f31dc5a1-ea54-4858-ac4c-2b990192ee4c-kube-api-access-ns6dn\") pod \"e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22\" (UID: \"f31dc5a1-ea54-4858-ac4c-2b990192ee4c\") " pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22" Nov 02 01:45:59.859212 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.859077 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f31dc5a1-ea54-4858-ac4c-2b990192ee4c-workload-socket\") pod \"e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22\" (UID: \"f31dc5a1-ea54-4858-ac4c-2b990192ee4c\") " pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22" Nov 02 01:45:59.859212 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.859097 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f31dc5a1-ea54-4858-ac4c-2b990192ee4c-istiod-ca-cert\") pod \"e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22\" (UID: \"f31dc5a1-ea54-4858-ac4c-2b990192ee4c\") " pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22" Nov 02 01:45:59.859212 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.859125 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f31dc5a1-ea54-4858-ac4c-2b990192ee4c-credential-socket\") pod \"e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22\" (UID: \"f31dc5a1-ea54-4858-ac4c-2b990192ee4c\") " pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22" Nov 02 01:45:59.859398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.859253 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f31dc5a1-ea54-4858-ac4c-2b990192ee4c-istio-data\") pod \"e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22\" (UID: \"f31dc5a1-ea54-4858-ac4c-2b990192ee4c\") " pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22" Nov 02 01:45:59.859482 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.859463 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f31dc5a1-ea54-4858-ac4c-2b990192ee4c-istio-token\") pod \"e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22\" (UID: \"f31dc5a1-ea54-4858-ac4c-2b990192ee4c\") " pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22" Nov 02 01:45:59.961150 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.961111 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f31dc5a1-ea54-4858-ac4c-2b990192ee4c-istio-podinfo\") pod \"e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22\" (UID: \"f31dc5a1-ea54-4858-ac4c-2b990192ee4c\") " pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22" Nov 02 01:45:59.961352 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.961212 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f31dc5a1-ea54-4858-ac4c-2b990192ee4c-workload-certs\") pod \"e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22\" (UID: \"f31dc5a1-ea54-4858-ac4c-2b990192ee4c\") " pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22" Nov 02 01:45:59.961352 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.961239 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns6dn\" (UniqueName: \"kubernetes.io/projected/f31dc5a1-ea54-4858-ac4c-2b990192ee4c-kube-api-access-ns6dn\") pod \"e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22\" (UID: \"f31dc5a1-ea54-4858-ac4c-2b990192ee4c\") " pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22" Nov 02 01:45:59.961352 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.961327 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f31dc5a1-ea54-4858-ac4c-2b990192ee4c-workload-socket\") pod \"e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22\" (UID: \"f31dc5a1-ea54-4858-ac4c-2b990192ee4c\") " pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22" Nov 02 01:45:59.961784 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.961759 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f31dc5a1-ea54-4858-ac4c-2b990192ee4c-workload-certs\") pod \"e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22\" (UID: \"f31dc5a1-ea54-4858-ac4c-2b990192ee4c\") " pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22" Nov 02 01:45:59.961898 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.961771 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f31dc5a1-ea54-4858-ac4c-2b990192ee4c-workload-socket\") pod \"e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22\" (UID: \"f31dc5a1-ea54-4858-ac4c-2b990192ee4c\") " pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22" Nov 02 01:45:59.961898 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.961796 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f31dc5a1-ea54-4858-ac4c-2b990192ee4c-istiod-ca-cert\") pod \"e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22\" (UID: \"f31dc5a1-ea54-4858-ac4c-2b990192ee4c\") " pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22" Nov 02 01:45:59.961898 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.961868 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f31dc5a1-ea54-4858-ac4c-2b990192ee4c-credential-socket\") pod \"e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22\" (UID: \"f31dc5a1-ea54-4858-ac4c-2b990192ee4c\") " pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22" Nov 02 01:45:59.962233 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.961910 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f31dc5a1-ea54-4858-ac4c-2b990192ee4c-istio-data\") pod \"e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22\" (UID: \"f31dc5a1-ea54-4858-ac4c-2b990192ee4c\") " pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22" Nov 02 01:45:59.962233 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.962067 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f31dc5a1-ea54-4858-ac4c-2b990192ee4c-istio-token\") pod \"e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22\" (UID: \"f31dc5a1-ea54-4858-ac4c-2b990192ee4c\") " pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22" Nov 02 01:45:59.962233 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.962163 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f31dc5a1-ea54-4858-ac4c-2b990192ee4c-credential-socket\") pod \"e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22\" (UID: \"f31dc5a1-ea54-4858-ac4c-2b990192ee4c\") " pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22" Nov 02 01:45:59.962452 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.962281 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f31dc5a1-ea54-4858-ac4c-2b990192ee4c-istio-data\") pod \"e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22\" (UID: \"f31dc5a1-ea54-4858-ac4c-2b990192ee4c\") " pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22" Nov 02 01:45:59.962452 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.962320 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f31dc5a1-ea54-4858-ac4c-2b990192ee4c-istiod-ca-cert\") pod \"e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22\" (UID: \"f31dc5a1-ea54-4858-ac4c-2b990192ee4c\") " pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22" Nov 02 01:45:59.962452 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.962100 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f31dc5a1-ea54-4858-ac4c-2b990192ee4c-istio-envoy\") pod \"e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22\" (UID: \"f31dc5a1-ea54-4858-ac4c-2b990192ee4c\") " pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22" Nov 02 01:45:59.964448 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.964423 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f31dc5a1-ea54-4858-ac4c-2b990192ee4c-istio-podinfo\") pod \"e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22\" (UID: \"f31dc5a1-ea54-4858-ac4c-2b990192ee4c\") " pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22" Nov 02 01:45:59.964684 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.964666 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f31dc5a1-ea54-4858-ac4c-2b990192ee4c-istio-envoy\") pod \"e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22\" (UID: \"f31dc5a1-ea54-4858-ac4c-2b990192ee4c\") " pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22" Nov 02 01:45:59.973948 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.973919 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns6dn\" (UniqueName: \"kubernetes.io/projected/f31dc5a1-ea54-4858-ac4c-2b990192ee4c-kube-api-access-ns6dn\") pod \"e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22\" (UID: \"f31dc5a1-ea54-4858-ac4c-2b990192ee4c\") " pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22" Nov 02 01:45:59.974149 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:45:59.974132 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f31dc5a1-ea54-4858-ac4c-2b990192ee4c-istio-token\") pod \"e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22\" (UID: \"f31dc5a1-ea54-4858-ac4c-2b990192ee4c\") " pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22" Nov 02 01:46:00.130172 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:00.130144 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22" Nov 02 01:46:00.425283 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:46:00.425251 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf31dc5a1_ea54_4858_ac4c_2b990192ee4c.slice/crio-40bc9baa4287766a5917407c753f37769ed9b41e8f2f938d5d1b7bb0eb1ba25d WatchSource:0}: Error finding container 40bc9baa4287766a5917407c753f37769ed9b41e8f2f938d5d1b7bb0eb1ba25d: Status 404 returned error can't find the container with id 40bc9baa4287766a5917407c753f37769ed9b41e8f2f938d5d1b7bb0eb1ba25d Nov 02 01:46:00.425620 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:00.425596 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22"] Nov 02 01:46:00.428556 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:00.428517 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:46:00.428667 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:00.428600 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:46:00.428667 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:00.428643 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:46:00.918295 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:00.918249 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22" event={"ID":"f31dc5a1-ea54-4858-ac4c-2b990192ee4c","Type":"ContainerStarted","Data":"b9a3277d2256fe4b6a16984a340fd19b7a21f7eb6724b4912989a4d1bba47422"} Nov 02 01:46:00.918857 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:00.918314 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22" event={"ID":"f31dc5a1-ea54-4858-ac4c-2b990192ee4c","Type":"ContainerStarted","Data":"40bc9baa4287766a5917407c753f37769ed9b41e8f2f938d5d1b7bb0eb1ba25d"} Nov 02 01:46:00.944546 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:00.944484 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22" podStartSLOduration=1.9444659340000001 podStartE2EDuration="1.944465934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:45:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:46:00.941044729 +0000 UTC m=+1865.562049941" watchObservedRunningTime="2026-11-02 01:46:00.944465934 +0000 UTC m=+1865.565471144" Nov 02 01:46:01.131309 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:01.131272 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22" Nov 02 01:46:01.133117 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:01.133090 4427 patch_prober.go:29] interesting pod/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Nov 02 01:46:01.133240 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:01.133138 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22" podUID="f31dc5a1-ea54-4858-ac4c-2b990192ee4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Nov 02 01:46:01.612529 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:46:01.612492 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:46:02.131184 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:02.131148 4427 patch_prober.go:29] interesting pod/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Nov 02 01:46:02.131594 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:02.131198 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22" podUID="f31dc5a1-ea54-4858-ac4c-2b990192ee4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Nov 02 01:46:02.656234 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:02.656193 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-00f1da-57d6bbd8d9-b8snz"] Nov 02 01:46:03.130766 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:03.130730 4427 patch_prober.go:29] interesting pod/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Nov 02 01:46:03.130961 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:03.130788 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22" podUID="f31dc5a1-ea54-4858-ac4c-2b990192ee4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Nov 02 01:46:04.130900 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:04.130862 4427 patch_prober.go:29] interesting pod/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Nov 02 01:46:04.131358 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:04.130921 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22" podUID="f31dc5a1-ea54-4858-ac4c-2b990192ee4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Nov 02 01:46:05.138404 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:05.135993 4427 patch_prober.go:29] interesting pod/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Nov 02 01:46:05.138404 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:05.136059 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22" podUID="f31dc5a1-ea54-4858-ac4c-2b990192ee4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Nov 02 01:46:06.130752 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:06.130718 4427 patch_prober.go:29] interesting pod/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Nov 02 01:46:06.130933 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:06.130771 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22" podUID="f31dc5a1-ea54-4858-ac4c-2b990192ee4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Nov 02 01:46:07.197219 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:07.197188 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22" Nov 02 01:46:07.197751 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:07.197513 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22" Nov 02 01:46:07.198695 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:07.198668 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22" Nov 02 01:46:07.657435 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:07.657351 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-00f1da-57d6bbd8d9-b8snz" podUID="9bf451ee-151b-46e9-91b7-27021d681c0d" containerName="payload-processing" containerID="cri-o://5c13de5ce0a03794e3177cf56776834f3de68d6ec32266efae880167d4d3ed48" gracePeriod=25 Nov 02 01:46:07.961605 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:07.961577 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-00f1da-57d6bbd8d9-b8snz" Nov 02 01:46:08.049806 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:08.049769 4427 generic.go:356] "Generic (PLEG): container finished" podID="9bf451ee-151b-46e9-91b7-27021d681c0d" containerID="5c13de5ce0a03794e3177cf56776834f3de68d6ec32266efae880167d4d3ed48" exitCode=0 Nov 02 01:46:08.050005 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:08.049821 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-00f1da-57d6bbd8d9-b8snz" Nov 02 01:46:08.050005 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:08.049858 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-00f1da-57d6bbd8d9-b8snz" event={"ID":"9bf451ee-151b-46e9-91b7-27021d681c0d","Type":"ContainerDied","Data":"5c13de5ce0a03794e3177cf56776834f3de68d6ec32266efae880167d4d3ed48"} Nov 02 01:46:08.050005 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:08.049931 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-00f1da-57d6bbd8d9-b8snz" event={"ID":"9bf451ee-151b-46e9-91b7-27021d681c0d","Type":"ContainerDied","Data":"45ac0a1de1b873b9d6a31b313661c911c432609f18b52487334294201e730115"} Nov 02 01:46:08.050005 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:08.049968 4427 scope.go:122] "RemoveContainer" containerID="5c13de5ce0a03794e3177cf56776834f3de68d6ec32266efae880167d4d3ed48" Nov 02 01:46:08.060537 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:08.060517 4427 scope.go:122] "RemoveContainer" containerID="5c13de5ce0a03794e3177cf56776834f3de68d6ec32266efae880167d4d3ed48" Nov 02 01:46:08.060852 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:46:08.060832 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c13de5ce0a03794e3177cf56776834f3de68d6ec32266efae880167d4d3ed48\": container with ID starting with 5c13de5ce0a03794e3177cf56776834f3de68d6ec32266efae880167d4d3ed48 not found: ID does not exist" containerID="5c13de5ce0a03794e3177cf56776834f3de68d6ec32266efae880167d4d3ed48" Nov 02 01:46:08.060897 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:08.060863 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c13de5ce0a03794e3177cf56776834f3de68d6ec32266efae880167d4d3ed48"} err="failed to get container status \"5c13de5ce0a03794e3177cf56776834f3de68d6ec32266efae880167d4d3ed48\": rpc error: code = NotFound desc = could not find container \"5c13de5ce0a03794e3177cf56776834f3de68d6ec32266efae880167d4d3ed48\": container with ID starting with 5c13de5ce0a03794e3177cf56776834f3de68d6ec32266efae880167d4d3ed48 not found: ID does not exist" Nov 02 01:46:08.075452 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:08.075427 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bf451ee-151b-46e9-91b7-27021d681c0d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bf451ee-151b-46e9-91b7-27021d681c0d-plugins-config-volume\") pod \"9bf451ee-151b-46e9-91b7-27021d681c0d\" (UID: \"9bf451ee-151b-46e9-91b7-27021d681c0d\") " Nov 02 01:46:08.075586 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:08.075495 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bf451ee-151b-46e9-91b7-27021d681c0d-kube-api-access-bk5d4\" (UniqueName: \"kubernetes.io/projected/9bf451ee-151b-46e9-91b7-27021d681c0d-kube-api-access-bk5d4\") pod \"9bf451ee-151b-46e9-91b7-27021d681c0d\" (UID: \"9bf451ee-151b-46e9-91b7-27021d681c0d\") " Nov 02 01:46:08.075864 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:08.075837 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bf451ee-151b-46e9-91b7-27021d681c0d-plugins-config-volume" pod "9bf451ee-151b-46e9-91b7-27021d681c0d" (UID: "9bf451ee-151b-46e9-91b7-27021d681c0d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:46:08.076319 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:08.076299 4427 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bf451ee-151b-46e9-91b7-27021d681c0d-plugins-config-volume\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:08.077863 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:08.077840 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bf451ee-151b-46e9-91b7-27021d681c0d-kube-api-access-bk5d4" pod "9bf451ee-151b-46e9-91b7-27021d681c0d" (UID: "9bf451ee-151b-46e9-91b7-27021d681c0d"). InnerVolumeSpecName "kube-api-access-bk5d4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:46:08.177732 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:08.177649 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bk5d4\" (UniqueName: \"kubernetes.io/projected/9bf451ee-151b-46e9-91b7-27021d681c0d-kube-api-access-bk5d4\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:08.375102 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:08.375063 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-00f1da-57d6bbd8d9-b8snz"] Nov 02 01:46:08.378334 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:08.378300 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-00f1da-57d6bbd8d9-b8snz"] Nov 02 01:46:09.534811 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:09.534772 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bf451ee-151b-46e9-91b7-27021d681c0d" path="/var/lib/kubelet/pods/9bf451ee-151b-46e9-91b7-27021d681c0d/volumes" Nov 02 01:46:09.780676 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:09.780641 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-12e7cb-859c4c76df-xgbzt"] Nov 02 01:46:09.781987 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:09.781956 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12e7cb-859c4c76df-xgbzt" podUID="db04af92-365b-4d73-970a-1da99dc55a3a" containerName="maas-api" containerID="cri-o://7c43e9fbabab169d1ff473af0abd9611f089388556ec452c772de7a408003023" gracePeriod=30 Nov 02 01:46:10.252283 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:10.252244 4427 generic.go:356] "Generic (PLEG): container finished" podID="db04af92-365b-4d73-970a-1da99dc55a3a" containerID="7c43e9fbabab169d1ff473af0abd9611f089388556ec452c772de7a408003023" exitCode=0 Nov 02 01:46:10.252503 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:10.252433 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12e7cb-859c4c76df-xgbzt" event={"ID":"db04af92-365b-4d73-970a-1da99dc55a3a","Type":"ContainerDied","Data":"7c43e9fbabab169d1ff473af0abd9611f089388556ec452c772de7a408003023"} Nov 02 01:46:10.341944 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:10.341916 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12e7cb-859c4c76df-xgbzt" Nov 02 01:46:10.449579 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:10.449526 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db04af92-365b-4d73-970a-1da99dc55a3a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/db04af92-365b-4d73-970a-1da99dc55a3a-metrics-certs\") pod \"db04af92-365b-4d73-970a-1da99dc55a3a\" (UID: \"db04af92-365b-4d73-970a-1da99dc55a3a\") " Nov 02 01:46:10.449793 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:10.449623 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db04af92-365b-4d73-970a-1da99dc55a3a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db04af92-365b-4d73-970a-1da99dc55a3a-openshift-service-ca\") pod \"db04af92-365b-4d73-970a-1da99dc55a3a\" (UID: \"db04af92-365b-4d73-970a-1da99dc55a3a\") " Nov 02 01:46:10.449793 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:10.449694 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db04af92-365b-4d73-970a-1da99dc55a3a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db04af92-365b-4d73-970a-1da99dc55a3a-maas-api-tls\") pod \"db04af92-365b-4d73-970a-1da99dc55a3a\" (UID: \"db04af92-365b-4d73-970a-1da99dc55a3a\") " Nov 02 01:46:10.449925 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:10.449873 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db04af92-365b-4d73-970a-1da99dc55a3a-kube-api-access-2msh4\" (UniqueName: \"kubernetes.io/projected/db04af92-365b-4d73-970a-1da99dc55a3a-kube-api-access-2msh4\") pod \"db04af92-365b-4d73-970a-1da99dc55a3a\" (UID: \"db04af92-365b-4d73-970a-1da99dc55a3a\") " Nov 02 01:46:10.450536 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:10.450503 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db04af92-365b-4d73-970a-1da99dc55a3a-openshift-service-ca" pod "db04af92-365b-4d73-970a-1da99dc55a3a" (UID: "db04af92-365b-4d73-970a-1da99dc55a3a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:46:10.452766 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:10.452735 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db04af92-365b-4d73-970a-1da99dc55a3a-kube-api-access-2msh4" pod "db04af92-365b-4d73-970a-1da99dc55a3a" (UID: "db04af92-365b-4d73-970a-1da99dc55a3a"). InnerVolumeSpecName "kube-api-access-2msh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:46:10.453051 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:10.453027 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db04af92-365b-4d73-970a-1da99dc55a3a-metrics-certs" pod "db04af92-365b-4d73-970a-1da99dc55a3a" (UID: "db04af92-365b-4d73-970a-1da99dc55a3a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:46:10.453249 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:10.453219 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db04af92-365b-4d73-970a-1da99dc55a3a-maas-api-tls" pod "db04af92-365b-4d73-970a-1da99dc55a3a" (UID: "db04af92-365b-4d73-970a-1da99dc55a3a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:46:10.551324 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:10.551227 4427 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db04af92-365b-4d73-970a-1da99dc55a3a-metrics-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:10.551324 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:10.551260 4427 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db04af92-365b-4d73-970a-1da99dc55a3a-openshift-service-ca\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:10.551324 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:10.551274 4427 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db04af92-365b-4d73-970a-1da99dc55a3a-maas-api-tls\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:10.551324 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:10.551286 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2msh4\" (UniqueName: \"kubernetes.io/projected/db04af92-365b-4d73-970a-1da99dc55a3a-kube-api-access-2msh4\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:11.261161 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:11.261125 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12e7cb-859c4c76df-xgbzt" event={"ID":"db04af92-365b-4d73-970a-1da99dc55a3a","Type":"ContainerDied","Data":"d88e51f29c79a0b6137570f86a17d58c21711fc70969797ea058d1fb03ab2cf9"} Nov 02 01:46:11.261389 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:11.261196 4427 scope.go:122] "RemoveContainer" containerID="7c43e9fbabab169d1ff473af0abd9611f089388556ec452c772de7a408003023" Nov 02 01:46:11.261389 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:11.261305 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12e7cb-859c4c76df-xgbzt" Nov 02 01:46:11.290881 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:11.290842 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-12e7cb-859c4c76df-xgbzt"] Nov 02 01:46:11.294020 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:11.293989 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-12e7cb-859c4c76df-xgbzt"] Nov 02 01:46:11.534829 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:11.534744 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db04af92-365b-4d73-970a-1da99dc55a3a" path="/var/lib/kubelet/pods/db04af92-365b-4d73-970a-1da99dc55a3a/volumes" Nov 02 01:46:13.302772 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:13.302740 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5687bc-ltx69"] Nov 02 01:46:13.303546 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:13.303525 4427 cpu_manager.go:414] "containerMap: removing container" podUID="9bf451ee-151b-46e9-91b7-27021d681c0d" containerName="payload-processing" Nov 02 01:46:13.303546 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:13.303546 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bf451ee-151b-46e9-91b7-27021d681c0d" containerName="payload-processing" Nov 02 01:46:13.303696 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:13.303563 4427 cpu_manager.go:414] "containerMap: removing container" podUID="db04af92-365b-4d73-970a-1da99dc55a3a" containerName="maas-api" Nov 02 01:46:13.303696 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:13.303571 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db04af92-365b-4d73-970a-1da99dc55a3a" containerName="maas-api" Nov 02 01:46:13.303803 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:13.303730 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bf451ee-151b-46e9-91b7-27021d681c0d" containerName="payload-processing" Nov 02 01:46:13.303803 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:13.303747 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="db04af92-365b-4d73-970a-1da99dc55a3a" containerName="maas-api" Nov 02 01:46:13.308755 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:13.308733 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5687bc-ltx69" Nov 02 01:46:13.313897 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:13.313808 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\"" Nov 02 01:46:13.314770 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:13.313916 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 01:46:13.317108 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:13.317080 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5687bc-ltx69"] Nov 02 01:46:13.392901 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:13.392863 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hntdx\" (UniqueName: \"kubernetes.io/projected/f70ac873-92f8-4223-b214-9ea1d5898075-kube-api-access-hntdx\") pod \"maas-api-revoke-keys-e2e-worker-w1-5687bc-ltx69\" (UID: \"f70ac873-92f8-4223-b214-9ea1d5898075\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5687bc-ltx69" Nov 02 01:46:13.393056 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:13.392982 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f70ac873-92f8-4223-b214-9ea1d5898075-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5687bc-ltx69\" (UID: \"f70ac873-92f8-4223-b214-9ea1d5898075\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5687bc-ltx69" Nov 02 01:46:13.393143 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:13.393127 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f70ac873-92f8-4223-b214-9ea1d5898075-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5687bc-ltx69\" (UID: \"f70ac873-92f8-4223-b214-9ea1d5898075\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5687bc-ltx69" Nov 02 01:46:13.494365 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:13.494322 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f70ac873-92f8-4223-b214-9ea1d5898075-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5687bc-ltx69\" (UID: \"f70ac873-92f8-4223-b214-9ea1d5898075\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5687bc-ltx69" Nov 02 01:46:13.494575 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:13.494519 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hntdx\" (UniqueName: \"kubernetes.io/projected/f70ac873-92f8-4223-b214-9ea1d5898075-kube-api-access-hntdx\") pod \"maas-api-revoke-keys-e2e-worker-w1-5687bc-ltx69\" (UID: \"f70ac873-92f8-4223-b214-9ea1d5898075\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5687bc-ltx69" Nov 02 01:46:13.494643 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:13.494610 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f70ac873-92f8-4223-b214-9ea1d5898075-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5687bc-ltx69\" (UID: \"f70ac873-92f8-4223-b214-9ea1d5898075\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5687bc-ltx69" Nov 02 01:46:13.495280 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:13.495252 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f70ac873-92f8-4223-b214-9ea1d5898075-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5687bc-ltx69\" (UID: \"f70ac873-92f8-4223-b214-9ea1d5898075\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5687bc-ltx69" Nov 02 01:46:13.496968 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:13.496939 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f70ac873-92f8-4223-b214-9ea1d5898075-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5687bc-ltx69\" (UID: \"f70ac873-92f8-4223-b214-9ea1d5898075\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5687bc-ltx69" Nov 02 01:46:13.523138 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:13.523061 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hntdx\" (UniqueName: \"kubernetes.io/projected/f70ac873-92f8-4223-b214-9ea1d5898075-kube-api-access-hntdx\") pod \"maas-api-revoke-keys-e2e-worker-w1-5687bc-ltx69\" (UID: \"f70ac873-92f8-4223-b214-9ea1d5898075\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5687bc-ltx69" Nov 02 01:46:13.625348 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:13.625311 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5687bc-ltx69" Nov 02 01:46:13.911145 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:13.911117 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5687bc-ltx69"] Nov 02 01:46:13.912151 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:46:13.912113 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf70ac873_92f8_4223_b214_9ea1d5898075.slice/crio-b79714d663d8148be56483affe7270c17ac2eebdd220d23cc7fa6da2a8ce9db4 WatchSource:0}: Error finding container b79714d663d8148be56483affe7270c17ac2eebdd220d23cc7fa6da2a8ce9db4: Status 404 returned error can't find the container with id b79714d663d8148be56483affe7270c17ac2eebdd220d23cc7fa6da2a8ce9db4 Nov 02 01:46:14.286448 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:14.286409 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5687bc-ltx69" event={"ID":"f70ac873-92f8-4223-b214-9ea1d5898075","Type":"ContainerStarted","Data":"3cb32bdf67dd76179964c43a8a2dc21607db46c936c34641f677737d36a0e7b5"} Nov 02 01:46:14.286620 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:14.286470 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5687bc-ltx69" event={"ID":"f70ac873-92f8-4223-b214-9ea1d5898075","Type":"ContainerStarted","Data":"b79714d663d8148be56483affe7270c17ac2eebdd220d23cc7fa6da2a8ce9db4"} Nov 02 01:46:14.309292 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:14.309238 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5687bc-ltx69" podStartSLOduration=1.309225805 podStartE2EDuration="1.309225805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:46:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:46:14.305295798 +0000 UTC m=+1878.926301008" watchObservedRunningTime="2026-11-02 01:46:14.309225805 +0000 UTC m=+1878.930231016" Nov 02 01:46:15.294821 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:15.294788 4427 generic.go:356] "Generic (PLEG): container finished" podID="f70ac873-92f8-4223-b214-9ea1d5898075" containerID="3cb32bdf67dd76179964c43a8a2dc21607db46c936c34641f677737d36a0e7b5" exitCode=0 Nov 02 01:46:15.295022 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:15.294873 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5687bc-ltx69" event={"ID":"f70ac873-92f8-4223-b214-9ea1d5898075","Type":"ContainerDied","Data":"3cb32bdf67dd76179964c43a8a2dc21607db46c936c34641f677737d36a0e7b5"} Nov 02 01:46:16.486947 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:16.486903 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5687bc-ltx69" Nov 02 01:46:16.539096 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:16.539060 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f70ac873-92f8-4223-b214-9ea1d5898075-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f70ac873-92f8-4223-b214-9ea1d5898075-maas-api-openshift-ca\") pod \"f70ac873-92f8-4223-b214-9ea1d5898075\" (UID: \"f70ac873-92f8-4223-b214-9ea1d5898075\") " Nov 02 01:46:16.539297 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:16.539210 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f70ac873-92f8-4223-b214-9ea1d5898075-kube-api-access-hntdx\" (UniqueName: \"kubernetes.io/projected/f70ac873-92f8-4223-b214-9ea1d5898075-kube-api-access-hntdx\") pod \"f70ac873-92f8-4223-b214-9ea1d5898075\" (UID: \"f70ac873-92f8-4223-b214-9ea1d5898075\") " Nov 02 01:46:16.539429 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:16.539412 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f70ac873-92f8-4223-b214-9ea1d5898075-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f70ac873-92f8-4223-b214-9ea1d5898075-maas-api-xks-ca\") pod \"f70ac873-92f8-4223-b214-9ea1d5898075\" (UID: \"f70ac873-92f8-4223-b214-9ea1d5898075\") " Nov 02 01:46:16.539522 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:16.539494 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f70ac873-92f8-4223-b214-9ea1d5898075-maas-api-openshift-ca" pod "f70ac873-92f8-4223-b214-9ea1d5898075" (UID: "f70ac873-92f8-4223-b214-9ea1d5898075"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:46:16.539917 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:16.539889 4427 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f70ac873-92f8-4223-b214-9ea1d5898075-maas-api-openshift-ca\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:16.542210 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:16.542173 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f70ac873-92f8-4223-b214-9ea1d5898075-kube-api-access-hntdx" pod "f70ac873-92f8-4223-b214-9ea1d5898075" (UID: "f70ac873-92f8-4223-b214-9ea1d5898075"). InnerVolumeSpecName "kube-api-access-hntdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:46:16.542583 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:16.542520 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f70ac873-92f8-4223-b214-9ea1d5898075-maas-api-xks-ca" pod "f70ac873-92f8-4223-b214-9ea1d5898075" (UID: "f70ac873-92f8-4223-b214-9ea1d5898075"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:46:16.642417 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:16.641420 4427 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f70ac873-92f8-4223-b214-9ea1d5898075-maas-api-xks-ca\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:16.642417 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:16.641457 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hntdx\" (UniqueName: \"kubernetes.io/projected/f70ac873-92f8-4223-b214-9ea1d5898075-kube-api-access-hntdx\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:17.311264 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:17.311237 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5687bc-ltx69" Nov 02 01:46:17.311514 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:17.311236 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5687bc-ltx69" event={"ID":"f70ac873-92f8-4223-b214-9ea1d5898075","Type":"ContainerDied","Data":"b79714d663d8148be56483affe7270c17ac2eebdd220d23cc7fa6da2a8ce9db4"} Nov 02 01:46:17.311514 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:17.311402 4427 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b79714d663d8148be56483affe7270c17ac2eebdd220d23cc7fa6da2a8ce9db4" Nov 02 01:46:17.842793 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:17.842756 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n"] Nov 02 01:46:17.843325 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:17.843028 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n" podUID="331a730d-c524-4f68-9360-25e29df1ffb4" containerName="istio-proxy" containerID="cri-o://337ba89cb083e16b593a211c66c84d79bf4ba3e895b894306f4a94e4f29e00db" gracePeriod=30 Nov 02 01:46:18.017408 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:18.015635 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr"] Nov 02 01:46:18.026395 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:18.025341 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks"] Nov 02 01:46:18.032398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:18.031172 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj"] Nov 02 01:46:18.051462 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:18.039709 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm"] Nov 02 01:46:18.604351 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:18.604309 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b9d816-tsqmh"] Nov 02 01:46:18.604907 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:18.604892 4427 cpu_manager.go:414] "containerMap: removing container" podUID="f70ac873-92f8-4223-b214-9ea1d5898075" containerName="revoke-keys" Nov 02 01:46:18.604967 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:18.604909 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f70ac873-92f8-4223-b214-9ea1d5898075" containerName="revoke-keys" Nov 02 01:46:18.605048 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:18.605038 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="f70ac873-92f8-4223-b214-9ea1d5898075" containerName="revoke-keys" Nov 02 01:46:18.608754 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:18.608726 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b9d816-tsqmh" Nov 02 01:46:18.622998 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:18.622953 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b9d816-tsqmh"] Nov 02 01:46:18.640193 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:18.640160 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 01:46:18.640426 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:18.640316 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\"" Nov 02 01:46:18.675423 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:18.675389 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8231498-d145-430c-872a-b05cb8718fc7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-b9d816-tsqmh\" (UID: \"b8231498-d145-430c-872a-b05cb8718fc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b9d816-tsqmh" Nov 02 01:46:18.675604 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:18.675585 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pksh8\" (UniqueName: \"kubernetes.io/projected/b8231498-d145-430c-872a-b05cb8718fc7-kube-api-access-pksh8\") pod \"maas-api-revoke-keys-e2e-worker-w6-b9d816-tsqmh\" (UID: \"b8231498-d145-430c-872a-b05cb8718fc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b9d816-tsqmh" Nov 02 01:46:18.675754 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:18.675739 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8231498-d145-430c-872a-b05cb8718fc7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-b9d816-tsqmh\" (UID: \"b8231498-d145-430c-872a-b05cb8718fc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b9d816-tsqmh" Nov 02 01:46:18.777608 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:18.777569 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pksh8\" (UniqueName: \"kubernetes.io/projected/b8231498-d145-430c-872a-b05cb8718fc7-kube-api-access-pksh8\") pod \"maas-api-revoke-keys-e2e-worker-w6-b9d816-tsqmh\" (UID: \"b8231498-d145-430c-872a-b05cb8718fc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b9d816-tsqmh" Nov 02 01:46:18.777785 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:18.777681 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8231498-d145-430c-872a-b05cb8718fc7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-b9d816-tsqmh\" (UID: \"b8231498-d145-430c-872a-b05cb8718fc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b9d816-tsqmh" Nov 02 01:46:18.777785 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:18.777753 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8231498-d145-430c-872a-b05cb8718fc7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-b9d816-tsqmh\" (UID: \"b8231498-d145-430c-872a-b05cb8718fc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b9d816-tsqmh" Nov 02 01:46:18.778473 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:18.778448 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8231498-d145-430c-872a-b05cb8718fc7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-b9d816-tsqmh\" (UID: \"b8231498-d145-430c-872a-b05cb8718fc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b9d816-tsqmh" Nov 02 01:46:18.780342 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:18.780317 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8231498-d145-430c-872a-b05cb8718fc7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-b9d816-tsqmh\" (UID: \"b8231498-d145-430c-872a-b05cb8718fc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b9d816-tsqmh" Nov 02 01:46:18.789703 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:18.789676 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pksh8\" (UniqueName: \"kubernetes.io/projected/b8231498-d145-430c-872a-b05cb8718fc7-kube-api-access-pksh8\") pod \"maas-api-revoke-keys-e2e-worker-w6-b9d816-tsqmh\" (UID: \"b8231498-d145-430c-872a-b05cb8718fc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b9d816-tsqmh" Nov 02 01:46:18.933592 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:18.933552 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b9d816-tsqmh" Nov 02 01:46:19.222458 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:19.222431 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b9d816-tsqmh"] Nov 02 01:46:19.224439 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:46:19.224405 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8231498_d145_430c_872a_b05cb8718fc7.slice/crio-25de86e02536168bb37cc2df6e8a6214ecc22eaa3782015e2bb85451deddadb4 WatchSource:0}: Error finding container 25de86e02536168bb37cc2df6e8a6214ecc22eaa3782015e2bb85451deddadb4: Status 404 returned error can't find the container with id 25de86e02536168bb37cc2df6e8a6214ecc22eaa3782015e2bb85451deddadb4 Nov 02 01:46:19.326635 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:19.326603 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b9d816-tsqmh" event={"ID":"b8231498-d145-430c-872a-b05cb8718fc7","Type":"ContainerStarted","Data":"25de86e02536168bb37cc2df6e8a6214ecc22eaa3782015e2bb85451deddadb4"} Nov 02 01:46:20.095439 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:20.095405 4427 patch_prober.go:29] interesting pod/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 01:46:20.095756 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:20.095462 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n" podUID="331a730d-c524-4f68-9360-25e29df1ffb4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 01:46:20.334682 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:20.334631 4427 generic.go:356] "Generic (PLEG): container finished" podID="b8231498-d145-430c-872a-b05cb8718fc7" containerID="6fdb2b1f725f2a69660139f20952de6e02c54930e2b68416f017210b49d20684" exitCode=0 Nov 02 01:46:20.334868 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:20.334709 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b9d816-tsqmh" event={"ID":"b8231498-d145-430c-872a-b05cb8718fc7","Type":"ContainerDied","Data":"6fdb2b1f725f2a69660139f20952de6e02c54930e2b68416f017210b49d20684"} Nov 02 01:46:21.506731 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:21.506704 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b9d816-tsqmh" Nov 02 01:46:21.614610 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:21.614582 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8231498-d145-430c-872a-b05cb8718fc7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8231498-d145-430c-872a-b05cb8718fc7-maas-api-openshift-ca\") pod \"b8231498-d145-430c-872a-b05cb8718fc7\" (UID: \"b8231498-d145-430c-872a-b05cb8718fc7\") " Nov 02 01:46:21.614788 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:21.614707 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8231498-d145-430c-872a-b05cb8718fc7-kube-api-access-pksh8\" (UniqueName: \"kubernetes.io/projected/b8231498-d145-430c-872a-b05cb8718fc7-kube-api-access-pksh8\") pod \"b8231498-d145-430c-872a-b05cb8718fc7\" (UID: \"b8231498-d145-430c-872a-b05cb8718fc7\") " Nov 02 01:46:21.614889 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:21.614795 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8231498-d145-430c-872a-b05cb8718fc7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8231498-d145-430c-872a-b05cb8718fc7-maas-api-xks-ca\") pod \"b8231498-d145-430c-872a-b05cb8718fc7\" (UID: \"b8231498-d145-430c-872a-b05cb8718fc7\") " Nov 02 01:46:21.615008 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:21.614972 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8231498-d145-430c-872a-b05cb8718fc7-maas-api-openshift-ca" pod "b8231498-d145-430c-872a-b05cb8718fc7" (UID: "b8231498-d145-430c-872a-b05cb8718fc7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:46:21.615285 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:21.615265 4427 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8231498-d145-430c-872a-b05cb8718fc7-maas-api-openshift-ca\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:21.617053 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:21.617026 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8231498-d145-430c-872a-b05cb8718fc7-kube-api-access-pksh8" pod "b8231498-d145-430c-872a-b05cb8718fc7" (UID: "b8231498-d145-430c-872a-b05cb8718fc7"). InnerVolumeSpecName "kube-api-access-pksh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:46:21.617170 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:21.617131 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8231498-d145-430c-872a-b05cb8718fc7-maas-api-xks-ca" pod "b8231498-d145-430c-872a-b05cb8718fc7" (UID: "b8231498-d145-430c-872a-b05cb8718fc7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:46:21.717014 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:21.716980 4427 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8231498-d145-430c-872a-b05cb8718fc7-maas-api-xks-ca\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:21.717014 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:21.717016 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pksh8\" (UniqueName: \"kubernetes.io/projected/b8231498-d145-430c-872a-b05cb8718fc7-kube-api-access-pksh8\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:22.350506 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:22.350473 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b9d816-tsqmh" Nov 02 01:46:22.350506 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:22.350494 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b9d816-tsqmh" event={"ID":"b8231498-d145-430c-872a-b05cb8718fc7","Type":"ContainerDied","Data":"25de86e02536168bb37cc2df6e8a6214ecc22eaa3782015e2bb85451deddadb4"} Nov 02 01:46:22.350716 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:22.350545 4427 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25de86e02536168bb37cc2df6e8a6214ecc22eaa3782015e2bb85451deddadb4" Nov 02 01:46:23.136338 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:23.136272 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n" Nov 02 01:46:23.238410 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:23.238330 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/331a730d-c524-4f68-9360-25e29df1ffb4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/331a730d-c524-4f68-9360-25e29df1ffb4-credential-socket\") pod \"331a730d-c524-4f68-9360-25e29df1ffb4\" (UID: \"331a730d-c524-4f68-9360-25e29df1ffb4\") " Nov 02 01:46:23.238619 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:23.238470 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/331a730d-c524-4f68-9360-25e29df1ffb4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/331a730d-c524-4f68-9360-25e29df1ffb4-istio-podinfo\") pod \"331a730d-c524-4f68-9360-25e29df1ffb4\" (UID: \"331a730d-c524-4f68-9360-25e29df1ffb4\") " Nov 02 01:46:23.238619 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:23.238512 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/331a730d-c524-4f68-9360-25e29df1ffb4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/331a730d-c524-4f68-9360-25e29df1ffb4-istiod-ca-cert\") pod \"331a730d-c524-4f68-9360-25e29df1ffb4\" (UID: \"331a730d-c524-4f68-9360-25e29df1ffb4\") " Nov 02 01:46:23.238619 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:23.238568 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/331a730d-c524-4f68-9360-25e29df1ffb4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/331a730d-c524-4f68-9360-25e29df1ffb4-workload-socket\") pod \"331a730d-c524-4f68-9360-25e29df1ffb4\" (UID: \"331a730d-c524-4f68-9360-25e29df1ffb4\") " Nov 02 01:46:23.238796 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:23.238665 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/331a730d-c524-4f68-9360-25e29df1ffb4-kube-api-access-fl9c4\" (UniqueName: \"kubernetes.io/projected/331a730d-c524-4f68-9360-25e29df1ffb4-kube-api-access-fl9c4\") pod \"331a730d-c524-4f68-9360-25e29df1ffb4\" (UID: \"331a730d-c524-4f68-9360-25e29df1ffb4\") " Nov 02 01:46:23.238796 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:23.238745 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/331a730d-c524-4f68-9360-25e29df1ffb4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/331a730d-c524-4f68-9360-25e29df1ffb4-workload-certs\") pod \"331a730d-c524-4f68-9360-25e29df1ffb4\" (UID: \"331a730d-c524-4f68-9360-25e29df1ffb4\") " Nov 02 01:46:23.238796 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:23.238788 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/331a730d-c524-4f68-9360-25e29df1ffb4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/331a730d-c524-4f68-9360-25e29df1ffb4-istio-data\") pod \"331a730d-c524-4f68-9360-25e29df1ffb4\" (UID: \"331a730d-c524-4f68-9360-25e29df1ffb4\") " Nov 02 01:46:23.238960 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:23.238870 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/331a730d-c524-4f68-9360-25e29df1ffb4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/331a730d-c524-4f68-9360-25e29df1ffb4-istio-envoy\") pod \"331a730d-c524-4f68-9360-25e29df1ffb4\" (UID: \"331a730d-c524-4f68-9360-25e29df1ffb4\") " Nov 02 01:46:23.238960 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:23.238897 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/331a730d-c524-4f68-9360-25e29df1ffb4-istio-token\" (UniqueName: \"kubernetes.io/projected/331a730d-c524-4f68-9360-25e29df1ffb4-istio-token\") pod \"331a730d-c524-4f68-9360-25e29df1ffb4\" (UID: \"331a730d-c524-4f68-9360-25e29df1ffb4\") " Nov 02 01:46:23.240388 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:23.240335 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/331a730d-c524-4f68-9360-25e29df1ffb4-workload-socket" pod "331a730d-c524-4f68-9360-25e29df1ffb4" (UID: "331a730d-c524-4f68-9360-25e29df1ffb4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:46:23.240689 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:23.240659 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/331a730d-c524-4f68-9360-25e29df1ffb4-istiod-ca-cert" pod "331a730d-c524-4f68-9360-25e29df1ffb4" (UID: "331a730d-c524-4f68-9360-25e29df1ffb4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:46:23.240998 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:23.240747 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/331a730d-c524-4f68-9360-25e29df1ffb4-credential-socket" pod "331a730d-c524-4f68-9360-25e29df1ffb4" (UID: "331a730d-c524-4f68-9360-25e29df1ffb4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:46:23.241268 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:23.241227 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/331a730d-c524-4f68-9360-25e29df1ffb4-istio-data" pod "331a730d-c524-4f68-9360-25e29df1ffb4" (UID: "331a730d-c524-4f68-9360-25e29df1ffb4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:46:23.241268 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:23.241239 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/331a730d-c524-4f68-9360-25e29df1ffb4-workload-certs" pod "331a730d-c524-4f68-9360-25e29df1ffb4" (UID: "331a730d-c524-4f68-9360-25e29df1ffb4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:46:23.242285 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:23.242250 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/331a730d-c524-4f68-9360-25e29df1ffb4-istio-token" pod "331a730d-c524-4f68-9360-25e29df1ffb4" (UID: "331a730d-c524-4f68-9360-25e29df1ffb4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:46:23.242764 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:23.242738 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/331a730d-c524-4f68-9360-25e29df1ffb4-istio-podinfo" pod "331a730d-c524-4f68-9360-25e29df1ffb4" (UID: "331a730d-c524-4f68-9360-25e29df1ffb4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:46:23.243134 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:23.243114 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/331a730d-c524-4f68-9360-25e29df1ffb4-istio-envoy" pod "331a730d-c524-4f68-9360-25e29df1ffb4" (UID: "331a730d-c524-4f68-9360-25e29df1ffb4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:46:23.244061 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:23.244031 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/331a730d-c524-4f68-9360-25e29df1ffb4-kube-api-access-fl9c4" pod "331a730d-c524-4f68-9360-25e29df1ffb4" (UID: "331a730d-c524-4f68-9360-25e29df1ffb4"). InnerVolumeSpecName "kube-api-access-fl9c4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:46:23.324741 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:23.324705 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t"] Nov 02 01:46:23.341631 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:23.341598 4427 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/331a730d-c524-4f68-9360-25e29df1ffb4-credential-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:23.341631 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:23.341630 4427 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/331a730d-c524-4f68-9360-25e29df1ffb4-istio-podinfo\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:23.341866 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:23.341643 4427 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/331a730d-c524-4f68-9360-25e29df1ffb4-istiod-ca-cert\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:23.341866 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:23.341656 4427 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/331a730d-c524-4f68-9360-25e29df1ffb4-workload-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:23.341866 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:23.341668 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fl9c4\" (UniqueName: \"kubernetes.io/projected/331a730d-c524-4f68-9360-25e29df1ffb4-kube-api-access-fl9c4\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:23.341866 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:23.341683 4427 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/331a730d-c524-4f68-9360-25e29df1ffb4-workload-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:23.341866 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:23.341694 4427 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/331a730d-c524-4f68-9360-25e29df1ffb4-istio-data\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:23.341866 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:23.341706 4427 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/331a730d-c524-4f68-9360-25e29df1ffb4-istio-envoy\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:23.341866 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:23.341717 4427 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/331a730d-c524-4f68-9360-25e29df1ffb4-istio-token\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:23.359752 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:23.359717 4427 generic.go:356] "Generic (PLEG): container finished" podID="331a730d-c524-4f68-9360-25e29df1ffb4" containerID="337ba89cb083e16b593a211c66c84d79bf4ba3e895b894306f4a94e4f29e00db" exitCode=0 Nov 02 01:46:23.359898 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:23.359805 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n" Nov 02 01:46:23.359898 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:23.359799 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n" event={"ID":"331a730d-c524-4f68-9360-25e29df1ffb4","Type":"ContainerDied","Data":"337ba89cb083e16b593a211c66c84d79bf4ba3e895b894306f4a94e4f29e00db"} Nov 02 01:46:23.360192 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:23.359978 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n" event={"ID":"331a730d-c524-4f68-9360-25e29df1ffb4","Type":"ContainerDied","Data":"d84bd8ef7dc9f9d075dc58d9d1be3e2cff2fd7966cf092e0cda9c59fab6dc7cf"} Nov 02 01:46:23.360192 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:23.360036 4427 scope.go:122] "RemoveContainer" containerID="337ba89cb083e16b593a211c66c84d79bf4ba3e895b894306f4a94e4f29e00db" Nov 02 01:46:23.375212 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:23.375191 4427 scope.go:122] "RemoveContainer" containerID="57486b9e30ae367c16b1c917c510c2aad33b96025e35d27ba63d680d0d2e82ed" Nov 02 01:46:23.391798 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:23.391767 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n"] Nov 02 01:46:23.391930 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:23.391904 4427 scope.go:122] "RemoveContainer" containerID="337ba89cb083e16b593a211c66c84d79bf4ba3e895b894306f4a94e4f29e00db" Nov 02 01:46:23.392313 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:46:23.392288 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"337ba89cb083e16b593a211c66c84d79bf4ba3e895b894306f4a94e4f29e00db\": container with ID starting with 337ba89cb083e16b593a211c66c84d79bf4ba3e895b894306f4a94e4f29e00db not found: ID does not exist" containerID="337ba89cb083e16b593a211c66c84d79bf4ba3e895b894306f4a94e4f29e00db" Nov 02 01:46:23.392436 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:23.392323 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"337ba89cb083e16b593a211c66c84d79bf4ba3e895b894306f4a94e4f29e00db"} err="failed to get container status \"337ba89cb083e16b593a211c66c84d79bf4ba3e895b894306f4a94e4f29e00db\": rpc error: code = NotFound desc = could not find container \"337ba89cb083e16b593a211c66c84d79bf4ba3e895b894306f4a94e4f29e00db\": container with ID starting with 337ba89cb083e16b593a211c66c84d79bf4ba3e895b894306f4a94e4f29e00db not found: ID does not exist" Nov 02 01:46:23.392436 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:23.392345 4427 scope.go:122] "RemoveContainer" containerID="57486b9e30ae367c16b1c917c510c2aad33b96025e35d27ba63d680d0d2e82ed" Nov 02 01:46:23.392764 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:46:23.392734 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57486b9e30ae367c16b1c917c510c2aad33b96025e35d27ba63d680d0d2e82ed\": container with ID starting with 57486b9e30ae367c16b1c917c510c2aad33b96025e35d27ba63d680d0d2e82ed not found: ID does not exist" containerID="57486b9e30ae367c16b1c917c510c2aad33b96025e35d27ba63d680d0d2e82ed" Nov 02 01:46:23.392895 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:23.392772 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57486b9e30ae367c16b1c917c510c2aad33b96025e35d27ba63d680d0d2e82ed"} err="failed to get container status \"57486b9e30ae367c16b1c917c510c2aad33b96025e35d27ba63d680d0d2e82ed\": rpc error: code = NotFound desc = could not find container \"57486b9e30ae367c16b1c917c510c2aad33b96025e35d27ba63d680d0d2e82ed\": container with ID starting with 57486b9e30ae367c16b1c917c510c2aad33b96025e35d27ba63d680d0d2e82ed not found: ID does not exist" Nov 02 01:46:23.401366 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:23.401341 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-12e7cb-openshift-default-6d8f5fd549-cw22n"] Nov 02 01:46:23.535137 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:23.535104 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="331a730d-c524-4f68-9360-25e29df1ffb4" path="/var/lib/kubelet/pods/331a730d-c524-4f68-9360-25e29df1ffb4/volumes" Nov 02 01:46:29.104643 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:29.104607 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-768858fd-hfnqf"] Nov 02 01:46:29.105521 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:29.105498 4427 cpu_manager.go:414] "containerMap: removing container" podUID="331a730d-c524-4f68-9360-25e29df1ffb4" containerName="istio-proxy" Nov 02 01:46:29.105521 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:29.105518 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="331a730d-c524-4f68-9360-25e29df1ffb4" containerName="istio-proxy" Nov 02 01:46:29.105664 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:29.105571 4427 cpu_manager.go:414] "containerMap: removing container" podUID="b8231498-d145-430c-872a-b05cb8718fc7" containerName="revoke-keys" Nov 02 01:46:29.105664 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:29.105579 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8231498-d145-430c-872a-b05cb8718fc7" containerName="revoke-keys" Nov 02 01:46:29.105772 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:29.105757 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8231498-d145-430c-872a-b05cb8718fc7" containerName="revoke-keys" Nov 02 01:46:29.105818 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:29.105780 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="331a730d-c524-4f68-9360-25e29df1ffb4" containerName="istio-proxy" Nov 02 01:46:29.105818 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:29.105792 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="331a730d-c524-4f68-9360-25e29df1ffb4" containerName="istio-proxy" Nov 02 01:46:29.111878 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:29.111856 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-768858fd-hfnqf" Nov 02 01:46:29.116239 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:29.116212 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-768858fd-hfnqf"] Nov 02 01:46:29.122887 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:29.122865 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 01:46:29.123023 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:29.122994 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\"" Nov 02 01:46:29.221554 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:29.221521 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c58e25b-d880-4738-98fc-2c7948a6153f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-768858fd-hfnqf\" (UID: \"1c58e25b-d880-4738-98fc-2c7948a6153f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-768858fd-hfnqf" Nov 02 01:46:29.221738 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:29.221596 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9d2c\" (UniqueName: \"kubernetes.io/projected/1c58e25b-d880-4738-98fc-2c7948a6153f-kube-api-access-r9d2c\") pod \"maas-api-revoke-sub-768858fd-hfnqf\" (UID: \"1c58e25b-d880-4738-98fc-2c7948a6153f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-768858fd-hfnqf" Nov 02 01:46:29.221738 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:29.221715 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c58e25b-d880-4738-98fc-2c7948a6153f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-768858fd-hfnqf\" (UID: \"1c58e25b-d880-4738-98fc-2c7948a6153f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-768858fd-hfnqf" Nov 02 01:46:29.323195 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:29.323164 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c58e25b-d880-4738-98fc-2c7948a6153f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-768858fd-hfnqf\" (UID: \"1c58e25b-d880-4738-98fc-2c7948a6153f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-768858fd-hfnqf" Nov 02 01:46:29.323418 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:29.323236 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9d2c\" (UniqueName: \"kubernetes.io/projected/1c58e25b-d880-4738-98fc-2c7948a6153f-kube-api-access-r9d2c\") pod \"maas-api-revoke-sub-768858fd-hfnqf\" (UID: \"1c58e25b-d880-4738-98fc-2c7948a6153f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-768858fd-hfnqf" Nov 02 01:46:29.323418 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:29.323296 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c58e25b-d880-4738-98fc-2c7948a6153f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-768858fd-hfnqf\" (UID: \"1c58e25b-d880-4738-98fc-2c7948a6153f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-768858fd-hfnqf" Nov 02 01:46:29.323899 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:29.323875 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c58e25b-d880-4738-98fc-2c7948a6153f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-768858fd-hfnqf\" (UID: \"1c58e25b-d880-4738-98fc-2c7948a6153f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-768858fd-hfnqf" Nov 02 01:46:29.325883 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:29.325862 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c58e25b-d880-4738-98fc-2c7948a6153f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-768858fd-hfnqf\" (UID: \"1c58e25b-d880-4738-98fc-2c7948a6153f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-768858fd-hfnqf" Nov 02 01:46:29.337654 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:29.337626 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9d2c\" (UniqueName: \"kubernetes.io/projected/1c58e25b-d880-4738-98fc-2c7948a6153f-kube-api-access-r9d2c\") pod \"maas-api-revoke-sub-768858fd-hfnqf\" (UID: \"1c58e25b-d880-4738-98fc-2c7948a6153f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-768858fd-hfnqf" Nov 02 01:46:29.436567 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:29.436512 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-768858fd-hfnqf" Nov 02 01:46:29.740889 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:46:29.740853 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c58e25b_d880_4738_98fc_2c7948a6153f.slice/crio-0e7d286d903db324cd9aa61768c88865317999c3013f796e79b0ef0f36dab541 WatchSource:0}: Error finding container 0e7d286d903db324cd9aa61768c88865317999c3013f796e79b0ef0f36dab541: Status 404 returned error can't find the container with id 0e7d286d903db324cd9aa61768c88865317999c3013f796e79b0ef0f36dab541 Nov 02 01:46:29.741024 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:29.741004 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-768858fd-hfnqf"] Nov 02 01:46:30.410402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:30.410347 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-768858fd-hfnqf" event={"ID":"1c58e25b-d880-4738-98fc-2c7948a6153f","Type":"ContainerStarted","Data":"b9341ce379d985ebf88e474ca879ace52814c24ea3f1db4e267a7e514325e0d4"} Nov 02 01:46:30.410923 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:30.410425 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-768858fd-hfnqf" event={"ID":"1c58e25b-d880-4738-98fc-2c7948a6153f","Type":"ContainerStarted","Data":"0e7d286d903db324cd9aa61768c88865317999c3013f796e79b0ef0f36dab541"} Nov 02 01:46:30.430190 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:30.430128 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-768858fd-hfnqf" podStartSLOduration=1.430112986 podStartE2EDuration="1.430112986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:46:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:46:30.429023524 +0000 UTC m=+1895.050028734" watchObservedRunningTime="2026-11-02 01:46:30.430112986 +0000 UTC m=+1895.051118197" Nov 02 01:46:31.418012 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:31.417969 4427 generic.go:356] "Generic (PLEG): container finished" podID="1c58e25b-d880-4738-98fc-2c7948a6153f" containerID="b9341ce379d985ebf88e474ca879ace52814c24ea3f1db4e267a7e514325e0d4" exitCode=0 Nov 02 01:46:31.418497 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:31.418021 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-768858fd-hfnqf" event={"ID":"1c58e25b-d880-4738-98fc-2c7948a6153f","Type":"ContainerDied","Data":"b9341ce379d985ebf88e474ca879ace52814c24ea3f1db4e267a7e514325e0d4"} Nov 02 01:46:31.585745 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:46:31.585708 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:46:31.643908 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:31.643869 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-068f2e65-dc9v4"] Nov 02 01:46:31.644583 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:31.644559 4427 cpu_manager.go:414] "containerMap: removing container" podUID="331a730d-c524-4f68-9360-25e29df1ffb4" containerName="istio-proxy" Nov 02 01:46:31.644583 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:31.644583 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="331a730d-c524-4f68-9360-25e29df1ffb4" containerName="istio-proxy" Nov 02 01:46:31.648422 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:31.648360 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-068f2e65-dc9v4" Nov 02 01:46:31.663707 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:31.663676 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-068f2e65-dc9v4"] Nov 02 01:46:31.675996 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:31.675896 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93282bc6-f0c6-48ce-bfac-998647223d42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-068f2e65-dc9v4\" (UID: \"93282bc6-f0c6-48ce-bfac-998647223d42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-068f2e65-dc9v4" Nov 02 01:46:31.676186 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:31.676103 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93282bc6-f0c6-48ce-bfac-998647223d42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-068f2e65-dc9v4\" (UID: \"93282bc6-f0c6-48ce-bfac-998647223d42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-068f2e65-dc9v4" Nov 02 01:46:31.676186 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:31.676173 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq7fn\" (UniqueName: \"kubernetes.io/projected/93282bc6-f0c6-48ce-bfac-998647223d42-kube-api-access-jq7fn\") pod \"maas-api-revoke-sub-068f2e65-dc9v4\" (UID: \"93282bc6-f0c6-48ce-bfac-998647223d42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-068f2e65-dc9v4" Nov 02 01:46:31.777640 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:31.777606 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93282bc6-f0c6-48ce-bfac-998647223d42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-068f2e65-dc9v4\" (UID: \"93282bc6-f0c6-48ce-bfac-998647223d42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-068f2e65-dc9v4" Nov 02 01:46:31.777845 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:31.777675 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq7fn\" (UniqueName: \"kubernetes.io/projected/93282bc6-f0c6-48ce-bfac-998647223d42-kube-api-access-jq7fn\") pod \"maas-api-revoke-sub-068f2e65-dc9v4\" (UID: \"93282bc6-f0c6-48ce-bfac-998647223d42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-068f2e65-dc9v4" Nov 02 01:46:31.777845 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:31.777807 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93282bc6-f0c6-48ce-bfac-998647223d42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-068f2e65-dc9v4\" (UID: \"93282bc6-f0c6-48ce-bfac-998647223d42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-068f2e65-dc9v4" Nov 02 01:46:31.778451 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:31.778425 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93282bc6-f0c6-48ce-bfac-998647223d42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-068f2e65-dc9v4\" (UID: \"93282bc6-f0c6-48ce-bfac-998647223d42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-068f2e65-dc9v4" Nov 02 01:46:31.780620 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:31.780595 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93282bc6-f0c6-48ce-bfac-998647223d42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-068f2e65-dc9v4\" (UID: \"93282bc6-f0c6-48ce-bfac-998647223d42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-068f2e65-dc9v4" Nov 02 01:46:31.790501 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:31.790270 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq7fn\" (UniqueName: \"kubernetes.io/projected/93282bc6-f0c6-48ce-bfac-998647223d42-kube-api-access-jq7fn\") pod \"maas-api-revoke-sub-068f2e65-dc9v4\" (UID: \"93282bc6-f0c6-48ce-bfac-998647223d42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-068f2e65-dc9v4" Nov 02 01:46:31.967135 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:31.967041 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-068f2e65-dc9v4" Nov 02 01:46:32.286983 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:32.286951 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-068f2e65-dc9v4"] Nov 02 01:46:32.288191 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:46:32.288169 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93282bc6_f0c6_48ce_bfac_998647223d42.slice/crio-89670eaea2b1085a0abb59d8af5ca5b26a0860606a0925c7e24ed72becea60f5 WatchSource:0}: Error finding container 89670eaea2b1085a0abb59d8af5ca5b26a0860606a0925c7e24ed72becea60f5: Status 404 returned error can't find the container with id 89670eaea2b1085a0abb59d8af5ca5b26a0860606a0925c7e24ed72becea60f5 Nov 02 01:46:32.426446 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:32.426406 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-068f2e65-dc9v4" event={"ID":"93282bc6-f0c6-48ce-bfac-998647223d42","Type":"ContainerStarted","Data":"89670eaea2b1085a0abb59d8af5ca5b26a0860606a0925c7e24ed72becea60f5"} Nov 02 01:46:32.558877 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:32.558846 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-768858fd-hfnqf" Nov 02 01:46:32.601021 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:32.600983 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c58e25b-d880-4738-98fc-2c7948a6153f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c58e25b-d880-4738-98fc-2c7948a6153f-maas-api-xks-ca\") pod \"1c58e25b-d880-4738-98fc-2c7948a6153f\" (UID: \"1c58e25b-d880-4738-98fc-2c7948a6153f\") " Nov 02 01:46:32.601213 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:32.601138 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c58e25b-d880-4738-98fc-2c7948a6153f-kube-api-access-r9d2c\" (UniqueName: \"kubernetes.io/projected/1c58e25b-d880-4738-98fc-2c7948a6153f-kube-api-access-r9d2c\") pod \"1c58e25b-d880-4738-98fc-2c7948a6153f\" (UID: \"1c58e25b-d880-4738-98fc-2c7948a6153f\") " Nov 02 01:46:32.601293 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:32.601263 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c58e25b-d880-4738-98fc-2c7948a6153f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c58e25b-d880-4738-98fc-2c7948a6153f-maas-api-openshift-ca\") pod \"1c58e25b-d880-4738-98fc-2c7948a6153f\" (UID: \"1c58e25b-d880-4738-98fc-2c7948a6153f\") " Nov 02 01:46:32.603306 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:32.603272 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c58e25b-d880-4738-98fc-2c7948a6153f-maas-api-openshift-ca" pod "1c58e25b-d880-4738-98fc-2c7948a6153f" (UID: "1c58e25b-d880-4738-98fc-2c7948a6153f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:46:32.604035 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:32.604003 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c58e25b-d880-4738-98fc-2c7948a6153f-maas-api-xks-ca" pod "1c58e25b-d880-4738-98fc-2c7948a6153f" (UID: "1c58e25b-d880-4738-98fc-2c7948a6153f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:46:32.604035 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:32.604019 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c58e25b-d880-4738-98fc-2c7948a6153f-kube-api-access-r9d2c" pod "1c58e25b-d880-4738-98fc-2c7948a6153f" (UID: "1c58e25b-d880-4738-98fc-2c7948a6153f"). InnerVolumeSpecName "kube-api-access-r9d2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:46:32.703185 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:32.703155 4427 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c58e25b-d880-4738-98fc-2c7948a6153f-maas-api-xks-ca\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:32.703185 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:32.703183 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9d2c\" (UniqueName: \"kubernetes.io/projected/1c58e25b-d880-4738-98fc-2c7948a6153f-kube-api-access-r9d2c\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:32.703389 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:32.703192 4427 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c58e25b-d880-4738-98fc-2c7948a6153f-maas-api-openshift-ca\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:33.061976 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.061929 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" podUID="47a9069f-1ffa-4c83-be8b-5e62f1d3275a" containerName="main" containerID="cri-o://ca04f4563d40cbea08860abbc1311b826618414b16f7cda510c5697f052dcd92" gracePeriod=45 Nov 02 01:46:33.070199 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.070159 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" podUID="706df1bd-19cf-4ab9-9aaa-3c12c97ccf45" containerName="main" containerID="cri-o://01e63d12b58e032cd37fc5a34502a745c82a0b55ccec096b44dc22c39d2ad327" gracePeriod=45 Nov 02 01:46:33.079250 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.079219 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" podUID="366c78ee-4aaf-4b7c-84b4-17028538cb1f" containerName="main" containerID="cri-o://7487fc8f9274de406bc39360f24319f7e915f6b1e443d16558494a87ccada363" gracePeriod=45 Nov 02 01:46:33.090693 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.090635 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" podUID="fb533b5f-3525-454a-8ce2-ea9246194f28" containerName="main" containerID="cri-o://17ce9f7ce5468ae59b54a5179dda3e4bc18154a3422a8050dc36150a36580289" gracePeriod=45 Nov 02 01:46:33.291574 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.291532 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" podUID="706df1bd-19cf-4ab9-9aaa-3c12c97ccf45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": dial tcp 10.133.0.47:8000: connect: connection refused" Nov 02 01:46:33.385232 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.385195 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" Nov 02 01:46:33.410410 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.410340 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" Nov 02 01:46:33.418435 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.412939 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/366c78ee-4aaf-4b7c-84b4-17028538cb1f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/366c78ee-4aaf-4b7c-84b4-17028538cb1f-tmp-dir\") pod \"366c78ee-4aaf-4b7c-84b4-17028538cb1f\" (UID: \"366c78ee-4aaf-4b7c-84b4-17028538cb1f\") " Nov 02 01:46:33.418435 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.412990 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/366c78ee-4aaf-4b7c-84b4-17028538cb1f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/366c78ee-4aaf-4b7c-84b4-17028538cb1f-dshm\") pod \"366c78ee-4aaf-4b7c-84b4-17028538cb1f\" (UID: \"366c78ee-4aaf-4b7c-84b4-17028538cb1f\") " Nov 02 01:46:33.418435 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.413084 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/366c78ee-4aaf-4b7c-84b4-17028538cb1f-home\" (UniqueName: \"kubernetes.io/empty-dir/366c78ee-4aaf-4b7c-84b4-17028538cb1f-home\") pod \"366c78ee-4aaf-4b7c-84b4-17028538cb1f\" (UID: \"366c78ee-4aaf-4b7c-84b4-17028538cb1f\") " Nov 02 01:46:33.418435 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.413113 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/366c78ee-4aaf-4b7c-84b4-17028538cb1f-tls-certs\" (UniqueName: \"kubernetes.io/secret/366c78ee-4aaf-4b7c-84b4-17028538cb1f-tls-certs\") pod \"366c78ee-4aaf-4b7c-84b4-17028538cb1f\" (UID: \"366c78ee-4aaf-4b7c-84b4-17028538cb1f\") " Nov 02 01:46:33.418435 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.413182 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/366c78ee-4aaf-4b7c-84b4-17028538cb1f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/366c78ee-4aaf-4b7c-84b4-17028538cb1f-model-cache\") pod \"366c78ee-4aaf-4b7c-84b4-17028538cb1f\" (UID: \"366c78ee-4aaf-4b7c-84b4-17028538cb1f\") " Nov 02 01:46:33.418435 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.413325 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/366c78ee-4aaf-4b7c-84b4-17028538cb1f-kube-api-access-kmgdv\" (UniqueName: \"kubernetes.io/projected/366c78ee-4aaf-4b7c-84b4-17028538cb1f-kube-api-access-kmgdv\") pod \"366c78ee-4aaf-4b7c-84b4-17028538cb1f\" (UID: \"366c78ee-4aaf-4b7c-84b4-17028538cb1f\") " Nov 02 01:46:33.418435 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.414468 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/366c78ee-4aaf-4b7c-84b4-17028538cb1f-home" pod "366c78ee-4aaf-4b7c-84b4-17028538cb1f" (UID: "366c78ee-4aaf-4b7c-84b4-17028538cb1f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:46:33.418435 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.414643 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/366c78ee-4aaf-4b7c-84b4-17028538cb1f-tmp-dir" pod "366c78ee-4aaf-4b7c-84b4-17028538cb1f" (UID: "366c78ee-4aaf-4b7c-84b4-17028538cb1f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:46:33.418435 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.415068 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/366c78ee-4aaf-4b7c-84b4-17028538cb1f-model-cache" pod "366c78ee-4aaf-4b7c-84b4-17028538cb1f" (UID: "366c78ee-4aaf-4b7c-84b4-17028538cb1f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:46:33.422230 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.421521 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/366c78ee-4aaf-4b7c-84b4-17028538cb1f-dshm" pod "366c78ee-4aaf-4b7c-84b4-17028538cb1f" (UID: "366c78ee-4aaf-4b7c-84b4-17028538cb1f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:46:33.422230 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.421553 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/366c78ee-4aaf-4b7c-84b4-17028538cb1f-kube-api-access-kmgdv" pod "366c78ee-4aaf-4b7c-84b4-17028538cb1f" (UID: "366c78ee-4aaf-4b7c-84b4-17028538cb1f"). InnerVolumeSpecName "kube-api-access-kmgdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:46:33.422230 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.421964 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/366c78ee-4aaf-4b7c-84b4-17028538cb1f-tls-certs" pod "366c78ee-4aaf-4b7c-84b4-17028538cb1f" (UID: "366c78ee-4aaf-4b7c-84b4-17028538cb1f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:46:33.438637 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.438603 4427 generic.go:356] "Generic (PLEG): container finished" podID="93282bc6-f0c6-48ce-bfac-998647223d42" containerID="90c6716482c2870b8f0fbbd748a3faea2cb9fd41f481ab5a32a0afc1fb0aeffd" exitCode=0 Nov 02 01:46:33.439078 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.438720 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-068f2e65-dc9v4" event={"ID":"93282bc6-f0c6-48ce-bfac-998647223d42","Type":"ContainerDied","Data":"90c6716482c2870b8f0fbbd748a3faea2cb9fd41f481ab5a32a0afc1fb0aeffd"} Nov 02 01:46:33.441172 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.441147 4427 generic.go:356] "Generic (PLEG): container finished" podID="706df1bd-19cf-4ab9-9aaa-3c12c97ccf45" containerID="01e63d12b58e032cd37fc5a34502a745c82a0b55ccec096b44dc22c39d2ad327" exitCode=0 Nov 02 01:46:33.441430 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.441289 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" event={"ID":"706df1bd-19cf-4ab9-9aaa-3c12c97ccf45","Type":"ContainerDied","Data":"01e63d12b58e032cd37fc5a34502a745c82a0b55ccec096b44dc22c39d2ad327"} Nov 02 01:46:33.442993 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.442973 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-768858fd-hfnqf" Nov 02 01:46:33.443109 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.442992 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-768858fd-hfnqf" event={"ID":"1c58e25b-d880-4738-98fc-2c7948a6153f","Type":"ContainerDied","Data":"0e7d286d903db324cd9aa61768c88865317999c3013f796e79b0ef0f36dab541"} Nov 02 01:46:33.443109 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.443040 4427 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e7d286d903db324cd9aa61768c88865317999c3013f796e79b0ef0f36dab541" Nov 02 01:46:33.447493 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.447466 4427 generic.go:356] "Generic (PLEG): container finished" podID="366c78ee-4aaf-4b7c-84b4-17028538cb1f" containerID="7487fc8f9274de406bc39360f24319f7e915f6b1e443d16558494a87ccada363" exitCode=0 Nov 02 01:46:33.447605 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.447517 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" Nov 02 01:46:33.447605 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.447574 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" event={"ID":"366c78ee-4aaf-4b7c-84b4-17028538cb1f","Type":"ContainerDied","Data":"7487fc8f9274de406bc39360f24319f7e915f6b1e443d16558494a87ccada363"} Nov 02 01:46:33.447728 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.447605 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks" event={"ID":"366c78ee-4aaf-4b7c-84b4-17028538cb1f","Type":"ContainerDied","Data":"bd3176f75d8edb89015f89c9b01237c740c67bb1bff83984a205d38b0419890a"} Nov 02 01:46:33.447728 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.447626 4427 scope.go:122] "RemoveContainer" containerID="7487fc8f9274de406bc39360f24319f7e915f6b1e443d16558494a87ccada363" Nov 02 01:46:33.450544 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.450523 4427 generic.go:356] "Generic (PLEG): container finished" podID="fb533b5f-3525-454a-8ce2-ea9246194f28" containerID="17ce9f7ce5468ae59b54a5179dda3e4bc18154a3422a8050dc36150a36580289" exitCode=0 Nov 02 01:46:33.456485 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.450591 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" event={"ID":"fb533b5f-3525-454a-8ce2-ea9246194f28","Type":"ContainerDied","Data":"17ce9f7ce5468ae59b54a5179dda3e4bc18154a3422a8050dc36150a36580289"} Nov 02 01:46:33.456485 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.453174 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" Nov 02 01:46:33.456485 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.456084 4427 generic.go:356] "Generic (PLEG): container finished" podID="47a9069f-1ffa-4c83-be8b-5e62f1d3275a" containerID="ca04f4563d40cbea08860abbc1311b826618414b16f7cda510c5697f052dcd92" exitCode=0 Nov 02 01:46:33.456485 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.456167 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" event={"ID":"47a9069f-1ffa-4c83-be8b-5e62f1d3275a","Type":"ContainerDied","Data":"ca04f4563d40cbea08860abbc1311b826618414b16f7cda510c5697f052dcd92"} Nov 02 01:46:33.456485 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.456186 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" event={"ID":"47a9069f-1ffa-4c83-be8b-5e62f1d3275a","Type":"ContainerDied","Data":"864650fa9ce6f6a15ee4f4288d5e200e94fb3038d6cf4a9633fa1311d2b458b3"} Nov 02 01:46:33.456485 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.456229 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr" Nov 02 01:46:33.475524 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.471284 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" Nov 02 01:46:33.475925 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.475901 4427 scope.go:122] "RemoveContainer" containerID="7487fc8f9274de406bc39360f24319f7e915f6b1e443d16558494a87ccada363" Nov 02 01:46:33.476335 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:46:33.476309 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7487fc8f9274de406bc39360f24319f7e915f6b1e443d16558494a87ccada363\": container with ID starting with 7487fc8f9274de406bc39360f24319f7e915f6b1e443d16558494a87ccada363 not found: ID does not exist" containerID="7487fc8f9274de406bc39360f24319f7e915f6b1e443d16558494a87ccada363" Nov 02 01:46:33.476774 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.476345 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7487fc8f9274de406bc39360f24319f7e915f6b1e443d16558494a87ccada363"} err="failed to get container status \"7487fc8f9274de406bc39360f24319f7e915f6b1e443d16558494a87ccada363\": rpc error: code = NotFound desc = could not find container \"7487fc8f9274de406bc39360f24319f7e915f6b1e443d16558494a87ccada363\": container with ID starting with 7487fc8f9274de406bc39360f24319f7e915f6b1e443d16558494a87ccada363 not found: ID does not exist" Nov 02 01:46:33.476774 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.476389 4427 scope.go:122] "RemoveContainer" containerID="ca04f4563d40cbea08860abbc1311b826618414b16f7cda510c5697f052dcd92" Nov 02 01:46:33.504430 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.504397 4427 scope.go:122] "RemoveContainer" containerID="ca04f4563d40cbea08860abbc1311b826618414b16f7cda510c5697f052dcd92" Nov 02 01:46:33.504793 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:46:33.504765 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca04f4563d40cbea08860abbc1311b826618414b16f7cda510c5697f052dcd92\": container with ID starting with ca04f4563d40cbea08860abbc1311b826618414b16f7cda510c5697f052dcd92 not found: ID does not exist" containerID="ca04f4563d40cbea08860abbc1311b826618414b16f7cda510c5697f052dcd92" Nov 02 01:46:33.504932 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.504800 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca04f4563d40cbea08860abbc1311b826618414b16f7cda510c5697f052dcd92"} err="failed to get container status \"ca04f4563d40cbea08860abbc1311b826618414b16f7cda510c5697f052dcd92\": rpc error: code = NotFound desc = could not find container \"ca04f4563d40cbea08860abbc1311b826618414b16f7cda510c5697f052dcd92\": container with ID starting with ca04f4563d40cbea08860abbc1311b826618414b16f7cda510c5697f052dcd92 not found: ID does not exist" Nov 02 01:46:33.514943 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.514906 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb533b5f-3525-454a-8ce2-ea9246194f28-tls-certs\" (UniqueName: \"kubernetes.io/secret/fb533b5f-3525-454a-8ce2-ea9246194f28-tls-certs\") pod \"fb533b5f-3525-454a-8ce2-ea9246194f28\" (UID: \"fb533b5f-3525-454a-8ce2-ea9246194f28\") " Nov 02 01:46:33.515065 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.514963 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47a9069f-1ffa-4c83-be8b-5e62f1d3275a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47a9069f-1ffa-4c83-be8b-5e62f1d3275a-tmp-dir\") pod \"47a9069f-1ffa-4c83-be8b-5e62f1d3275a\" (UID: \"47a9069f-1ffa-4c83-be8b-5e62f1d3275a\") " Nov 02 01:46:33.515065 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.514999 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/706df1bd-19cf-4ab9-9aaa-3c12c97ccf45-dshm\" (UniqueName: \"kubernetes.io/empty-dir/706df1bd-19cf-4ab9-9aaa-3c12c97ccf45-dshm\") pod \"706df1bd-19cf-4ab9-9aaa-3c12c97ccf45\" (UID: \"706df1bd-19cf-4ab9-9aaa-3c12c97ccf45\") " Nov 02 01:46:33.515065 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.515043 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/706df1bd-19cf-4ab9-9aaa-3c12c97ccf45-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/706df1bd-19cf-4ab9-9aaa-3c12c97ccf45-tmp-dir\") pod \"706df1bd-19cf-4ab9-9aaa-3c12c97ccf45\" (UID: \"706df1bd-19cf-4ab9-9aaa-3c12c97ccf45\") " Nov 02 01:46:33.515216 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.515079 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb533b5f-3525-454a-8ce2-ea9246194f28-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb533b5f-3525-454a-8ce2-ea9246194f28-tmp-dir\") pod \"fb533b5f-3525-454a-8ce2-ea9246194f28\" (UID: \"fb533b5f-3525-454a-8ce2-ea9246194f28\") " Nov 02 01:46:33.515252 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.515216 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47a9069f-1ffa-4c83-be8b-5e62f1d3275a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/47a9069f-1ffa-4c83-be8b-5e62f1d3275a-dshm\") pod \"47a9069f-1ffa-4c83-be8b-5e62f1d3275a\" (UID: \"47a9069f-1ffa-4c83-be8b-5e62f1d3275a\") " Nov 02 01:46:33.515302 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.515265 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/706df1bd-19cf-4ab9-9aaa-3c12c97ccf45-home\" (UniqueName: \"kubernetes.io/empty-dir/706df1bd-19cf-4ab9-9aaa-3c12c97ccf45-home\") pod \"706df1bd-19cf-4ab9-9aaa-3c12c97ccf45\" (UID: \"706df1bd-19cf-4ab9-9aaa-3c12c97ccf45\") " Nov 02 01:46:33.515350 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.515301 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb533b5f-3525-454a-8ce2-ea9246194f28-home\" (UniqueName: \"kubernetes.io/empty-dir/fb533b5f-3525-454a-8ce2-ea9246194f28-home\") pod \"fb533b5f-3525-454a-8ce2-ea9246194f28\" (UID: \"fb533b5f-3525-454a-8ce2-ea9246194f28\") " Nov 02 01:46:33.515454 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.515362 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47a9069f-1ffa-4c83-be8b-5e62f1d3275a-kube-api-access-z577g\" (UniqueName: \"kubernetes.io/projected/47a9069f-1ffa-4c83-be8b-5e62f1d3275a-kube-api-access-z577g\") pod \"47a9069f-1ffa-4c83-be8b-5e62f1d3275a\" (UID: \"47a9069f-1ffa-4c83-be8b-5e62f1d3275a\") " Nov 02 01:46:33.515521 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.515461 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/706df1bd-19cf-4ab9-9aaa-3c12c97ccf45-kube-api-access-clpz2\" (UniqueName: \"kubernetes.io/projected/706df1bd-19cf-4ab9-9aaa-3c12c97ccf45-kube-api-access-clpz2\") pod \"706df1bd-19cf-4ab9-9aaa-3c12c97ccf45\" (UID: \"706df1bd-19cf-4ab9-9aaa-3c12c97ccf45\") " Nov 02 01:46:33.515521 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.515498 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb533b5f-3525-454a-8ce2-ea9246194f28-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb533b5f-3525-454a-8ce2-ea9246194f28-model-cache\") pod \"fb533b5f-3525-454a-8ce2-ea9246194f28\" (UID: \"fb533b5f-3525-454a-8ce2-ea9246194f28\") " Nov 02 01:46:33.515623 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.515525 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47a9069f-1ffa-4c83-be8b-5e62f1d3275a-tls-certs\" (UniqueName: \"kubernetes.io/secret/47a9069f-1ffa-4c83-be8b-5e62f1d3275a-tls-certs\") pod \"47a9069f-1ffa-4c83-be8b-5e62f1d3275a\" (UID: \"47a9069f-1ffa-4c83-be8b-5e62f1d3275a\") " Nov 02 01:46:33.515623 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.515550 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47a9069f-1ffa-4c83-be8b-5e62f1d3275a-home\" (UniqueName: \"kubernetes.io/empty-dir/47a9069f-1ffa-4c83-be8b-5e62f1d3275a-home\") pod \"47a9069f-1ffa-4c83-be8b-5e62f1d3275a\" (UID: \"47a9069f-1ffa-4c83-be8b-5e62f1d3275a\") " Nov 02 01:46:33.515623 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.515582 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/706df1bd-19cf-4ab9-9aaa-3c12c97ccf45-tls-certs\" (UniqueName: \"kubernetes.io/secret/706df1bd-19cf-4ab9-9aaa-3c12c97ccf45-tls-certs\") pod \"706df1bd-19cf-4ab9-9aaa-3c12c97ccf45\" (UID: \"706df1bd-19cf-4ab9-9aaa-3c12c97ccf45\") " Nov 02 01:46:33.515806 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.515671 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb533b5f-3525-454a-8ce2-ea9246194f28-kube-api-access-74g7g\" (UniqueName: \"kubernetes.io/projected/fb533b5f-3525-454a-8ce2-ea9246194f28-kube-api-access-74g7g\") pod \"fb533b5f-3525-454a-8ce2-ea9246194f28\" (UID: \"fb533b5f-3525-454a-8ce2-ea9246194f28\") " Nov 02 01:46:33.515806 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.515708 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/706df1bd-19cf-4ab9-9aaa-3c12c97ccf45-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/706df1bd-19cf-4ab9-9aaa-3c12c97ccf45-model-cache\") pod \"706df1bd-19cf-4ab9-9aaa-3c12c97ccf45\" (UID: \"706df1bd-19cf-4ab9-9aaa-3c12c97ccf45\") " Nov 02 01:46:33.515806 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.515761 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb533b5f-3525-454a-8ce2-ea9246194f28-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb533b5f-3525-454a-8ce2-ea9246194f28-dshm\") pod \"fb533b5f-3525-454a-8ce2-ea9246194f28\" (UID: \"fb533b5f-3525-454a-8ce2-ea9246194f28\") " Nov 02 01:46:33.515940 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.515855 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47a9069f-1ffa-4c83-be8b-5e62f1d3275a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47a9069f-1ffa-4c83-be8b-5e62f1d3275a-model-cache\") pod \"47a9069f-1ffa-4c83-be8b-5e62f1d3275a\" (UID: \"47a9069f-1ffa-4c83-be8b-5e62f1d3275a\") " Nov 02 01:46:33.516503 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.516476 4427 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/366c78ee-4aaf-4b7c-84b4-17028538cb1f-home\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:33.516615 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.516507 4427 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/366c78ee-4aaf-4b7c-84b4-17028538cb1f-tls-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:33.516615 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.516522 4427 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/366c78ee-4aaf-4b7c-84b4-17028538cb1f-model-cache\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:33.516615 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.516538 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmgdv\" (UniqueName: \"kubernetes.io/projected/366c78ee-4aaf-4b7c-84b4-17028538cb1f-kube-api-access-kmgdv\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:33.516615 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.516552 4427 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/366c78ee-4aaf-4b7c-84b4-17028538cb1f-tmp-dir\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:33.516615 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.516567 4427 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/366c78ee-4aaf-4b7c-84b4-17028538cb1f-dshm\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:33.517144 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.517116 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/706df1bd-19cf-4ab9-9aaa-3c12c97ccf45-tmp-dir" pod "706df1bd-19cf-4ab9-9aaa-3c12c97ccf45" (UID: "706df1bd-19cf-4ab9-9aaa-3c12c97ccf45"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:46:33.517445 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.517282 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47a9069f-1ffa-4c83-be8b-5e62f1d3275a-home" pod "47a9069f-1ffa-4c83-be8b-5e62f1d3275a" (UID: "47a9069f-1ffa-4c83-be8b-5e62f1d3275a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:46:33.517609 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.517583 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47a9069f-1ffa-4c83-be8b-5e62f1d3275a-tmp-dir" pod "47a9069f-1ffa-4c83-be8b-5e62f1d3275a" (UID: "47a9069f-1ffa-4c83-be8b-5e62f1d3275a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:46:33.518028 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.517804 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/706df1bd-19cf-4ab9-9aaa-3c12c97ccf45-home" pod "706df1bd-19cf-4ab9-9aaa-3c12c97ccf45" (UID: "706df1bd-19cf-4ab9-9aaa-3c12c97ccf45"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:46:33.518339 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.518314 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/706df1bd-19cf-4ab9-9aaa-3c12c97ccf45-model-cache" pod "706df1bd-19cf-4ab9-9aaa-3c12c97ccf45" (UID: "706df1bd-19cf-4ab9-9aaa-3c12c97ccf45"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:46:33.518455 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.518335 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb533b5f-3525-454a-8ce2-ea9246194f28-home" pod "fb533b5f-3525-454a-8ce2-ea9246194f28" (UID: "fb533b5f-3525-454a-8ce2-ea9246194f28"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:46:33.520605 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.520579 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47a9069f-1ffa-4c83-be8b-5e62f1d3275a-model-cache" pod "47a9069f-1ffa-4c83-be8b-5e62f1d3275a" (UID: "47a9069f-1ffa-4c83-be8b-5e62f1d3275a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:46:33.521178 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.520739 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb533b5f-3525-454a-8ce2-ea9246194f28-tmp-dir" pod "fb533b5f-3525-454a-8ce2-ea9246194f28" (UID: "fb533b5f-3525-454a-8ce2-ea9246194f28"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:46:33.521178 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.520793 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb533b5f-3525-454a-8ce2-ea9246194f28-model-cache" pod "fb533b5f-3525-454a-8ce2-ea9246194f28" (UID: "fb533b5f-3525-454a-8ce2-ea9246194f28"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:46:33.521178 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.520859 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/706df1bd-19cf-4ab9-9aaa-3c12c97ccf45-dshm" pod "706df1bd-19cf-4ab9-9aaa-3c12c97ccf45" (UID: "706df1bd-19cf-4ab9-9aaa-3c12c97ccf45"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:46:33.522712 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.522592 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47a9069f-1ffa-4c83-be8b-5e62f1d3275a-kube-api-access-z577g" pod "47a9069f-1ffa-4c83-be8b-5e62f1d3275a" (UID: "47a9069f-1ffa-4c83-be8b-5e62f1d3275a"). InnerVolumeSpecName "kube-api-access-z577g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:46:33.523341 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.523305 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb533b5f-3525-454a-8ce2-ea9246194f28-dshm" pod "fb533b5f-3525-454a-8ce2-ea9246194f28" (UID: "fb533b5f-3525-454a-8ce2-ea9246194f28"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:46:33.524500 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.524450 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks"] Nov 02 01:46:33.526483 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.526443 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/706df1bd-19cf-4ab9-9aaa-3c12c97ccf45-kube-api-access-clpz2" pod "706df1bd-19cf-4ab9-9aaa-3c12c97ccf45" (UID: "706df1bd-19cf-4ab9-9aaa-3c12c97ccf45"). InnerVolumeSpecName "kube-api-access-clpz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:46:33.526718 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.526691 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47a9069f-1ffa-4c83-be8b-5e62f1d3275a-tls-certs" pod "47a9069f-1ffa-4c83-be8b-5e62f1d3275a" (UID: "47a9069f-1ffa-4c83-be8b-5e62f1d3275a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:46:33.526937 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.526910 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47a9069f-1ffa-4c83-be8b-5e62f1d3275a-dshm" pod "47a9069f-1ffa-4c83-be8b-5e62f1d3275a" (UID: "47a9069f-1ffa-4c83-be8b-5e62f1d3275a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:46:33.527162 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.527132 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/706df1bd-19cf-4ab9-9aaa-3c12c97ccf45-tls-certs" pod "706df1bd-19cf-4ab9-9aaa-3c12c97ccf45" (UID: "706df1bd-19cf-4ab9-9aaa-3c12c97ccf45"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:46:33.528104 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.528071 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb533b5f-3525-454a-8ce2-ea9246194f28-kube-api-access-74g7g" pod "fb533b5f-3525-454a-8ce2-ea9246194f28" (UID: "fb533b5f-3525-454a-8ce2-ea9246194f28"). InnerVolumeSpecName "kube-api-access-74g7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:46:33.529572 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.529539 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb533b5f-3525-454a-8ce2-ea9246194f28-tls-certs" pod "fb533b5f-3525-454a-8ce2-ea9246194f28" (UID: "fb533b5f-3525-454a-8ce2-ea9246194f28"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:46:33.541011 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.540984 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-5687bc/distinct-w1-5687bc-kserve-55887d7865-456ks"] Nov 02 01:46:33.618262 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.618222 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74g7g\" (UniqueName: \"kubernetes.io/projected/fb533b5f-3525-454a-8ce2-ea9246194f28-kube-api-access-74g7g\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:33.618474 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.618282 4427 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/706df1bd-19cf-4ab9-9aaa-3c12c97ccf45-model-cache\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:33.618474 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.618293 4427 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb533b5f-3525-454a-8ce2-ea9246194f28-dshm\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:33.618474 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.618303 4427 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47a9069f-1ffa-4c83-be8b-5e62f1d3275a-model-cache\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:33.618474 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.618312 4427 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb533b5f-3525-454a-8ce2-ea9246194f28-tls-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:33.618474 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.618321 4427 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47a9069f-1ffa-4c83-be8b-5e62f1d3275a-tmp-dir\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:33.618474 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.618329 4427 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/706df1bd-19cf-4ab9-9aaa-3c12c97ccf45-dshm\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:33.618474 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.618336 4427 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/706df1bd-19cf-4ab9-9aaa-3c12c97ccf45-tmp-dir\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:33.618474 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.618344 4427 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb533b5f-3525-454a-8ce2-ea9246194f28-tmp-dir\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:33.618474 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.618351 4427 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47a9069f-1ffa-4c83-be8b-5e62f1d3275a-dshm\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:33.618474 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.618358 4427 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/706df1bd-19cf-4ab9-9aaa-3c12c97ccf45-home\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:33.618474 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.618366 4427 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb533b5f-3525-454a-8ce2-ea9246194f28-home\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:33.618474 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.618394 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z577g\" (UniqueName: \"kubernetes.io/projected/47a9069f-1ffa-4c83-be8b-5e62f1d3275a-kube-api-access-z577g\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:33.618474 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.618406 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clpz2\" (UniqueName: \"kubernetes.io/projected/706df1bd-19cf-4ab9-9aaa-3c12c97ccf45-kube-api-access-clpz2\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:33.618474 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.618419 4427 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb533b5f-3525-454a-8ce2-ea9246194f28-model-cache\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:33.618474 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.618428 4427 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47a9069f-1ffa-4c83-be8b-5e62f1d3275a-tls-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:33.618474 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.618435 4427 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47a9069f-1ffa-4c83-be8b-5e62f1d3275a-home\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:33.618474 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.618444 4427 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/706df1bd-19cf-4ab9-9aaa-3c12c97ccf45-tls-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:33.782786 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.782750 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr"] Nov 02 01:46:33.788684 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:33.788655 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-5687bc/distinct-2-w1-5687bc-kserve-5b98845f8-9blxr"] Nov 02 01:46:34.399166 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:34.399118 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" podUID="fb533b5f-3525-454a-8ce2-ea9246194f28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:46:34.464853 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:34.464818 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" Nov 02 01:46:34.465267 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:34.464848 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj" event={"ID":"706df1bd-19cf-4ab9-9aaa-3c12c97ccf45","Type":"ContainerDied","Data":"70918b0a9d6f77b25ec2d3a9478580f935485c810429c95d51c18b20f9d9d06b"} Nov 02 01:46:34.465267 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:34.464904 4427 scope.go:122] "RemoveContainer" containerID="01e63d12b58e032cd37fc5a34502a745c82a0b55ccec096b44dc22c39d2ad327" Nov 02 01:46:34.468522 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:34.467857 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" Nov 02 01:46:34.468522 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:34.467853 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm" event={"ID":"fb533b5f-3525-454a-8ce2-ea9246194f28","Type":"ContainerDied","Data":"8de130c26b61edd8e7cc11278fa430ed16b0a0b6c68153b2370aafbd28c3e777"} Nov 02 01:46:34.475839 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:34.475813 4427 scope.go:122] "RemoveContainer" containerID="17ce9f7ce5468ae59b54a5179dda3e4bc18154a3422a8050dc36150a36580289" Nov 02 01:46:34.487609 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:34.487557 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj"] Nov 02 01:46:34.491313 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:34.491285 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-5687bc/premium-w1-5687bc-kserve-5b5d47dd98-m7vbj"] Nov 02 01:46:34.511049 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:34.509633 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm"] Nov 02 01:46:34.513455 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:34.513425 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-5687bc/simulator-w1-5687bc-kserve-75d664fc67-gclhm"] Nov 02 01:46:34.649633 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:34.649562 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-068f2e65-dc9v4" Nov 02 01:46:34.730984 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:34.730948 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93282bc6-f0c6-48ce-bfac-998647223d42-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93282bc6-f0c6-48ce-bfac-998647223d42-maas-api-openshift-ca\") pod \"93282bc6-f0c6-48ce-bfac-998647223d42\" (UID: \"93282bc6-f0c6-48ce-bfac-998647223d42\") " Nov 02 01:46:34.731183 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:34.731001 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93282bc6-f0c6-48ce-bfac-998647223d42-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93282bc6-f0c6-48ce-bfac-998647223d42-maas-api-xks-ca\") pod \"93282bc6-f0c6-48ce-bfac-998647223d42\" (UID: \"93282bc6-f0c6-48ce-bfac-998647223d42\") " Nov 02 01:46:34.731183 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:34.731069 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93282bc6-f0c6-48ce-bfac-998647223d42-kube-api-access-jq7fn\" (UniqueName: \"kubernetes.io/projected/93282bc6-f0c6-48ce-bfac-998647223d42-kube-api-access-jq7fn\") pod \"93282bc6-f0c6-48ce-bfac-998647223d42\" (UID: \"93282bc6-f0c6-48ce-bfac-998647223d42\") " Nov 02 01:46:34.731357 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:34.731331 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93282bc6-f0c6-48ce-bfac-998647223d42-maas-api-openshift-ca" pod "93282bc6-f0c6-48ce-bfac-998647223d42" (UID: "93282bc6-f0c6-48ce-bfac-998647223d42"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:46:34.731699 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:34.731671 4427 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93282bc6-f0c6-48ce-bfac-998647223d42-maas-api-openshift-ca\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:34.733309 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:34.733288 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93282bc6-f0c6-48ce-bfac-998647223d42-maas-api-xks-ca" pod "93282bc6-f0c6-48ce-bfac-998647223d42" (UID: "93282bc6-f0c6-48ce-bfac-998647223d42"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:46:34.733429 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:34.733352 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93282bc6-f0c6-48ce-bfac-998647223d42-kube-api-access-jq7fn" pod "93282bc6-f0c6-48ce-bfac-998647223d42" (UID: "93282bc6-f0c6-48ce-bfac-998647223d42"). InnerVolumeSpecName "kube-api-access-jq7fn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:46:34.833358 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:34.833322 4427 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93282bc6-f0c6-48ce-bfac-998647223d42-maas-api-xks-ca\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:34.833358 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:34.833358 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq7fn\" (UniqueName: \"kubernetes.io/projected/93282bc6-f0c6-48ce-bfac-998647223d42-kube-api-access-jq7fn\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:35.478122 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:35.478094 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-068f2e65-dc9v4" Nov 02 01:46:35.478576 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:35.478094 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-068f2e65-dc9v4" event={"ID":"93282bc6-f0c6-48ce-bfac-998647223d42","Type":"ContainerDied","Data":"89670eaea2b1085a0abb59d8af5ca5b26a0860606a0925c7e24ed72becea60f5"} Nov 02 01:46:35.478576 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:35.478245 4427 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89670eaea2b1085a0abb59d8af5ca5b26a0860606a0925c7e24ed72becea60f5" Nov 02 01:46:35.534404 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:35.534354 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="366c78ee-4aaf-4b7c-84b4-17028538cb1f" path="/var/lib/kubelet/pods/366c78ee-4aaf-4b7c-84b4-17028538cb1f/volumes" Nov 02 01:46:35.534911 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:35.534892 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47a9069f-1ffa-4c83-be8b-5e62f1d3275a" path="/var/lib/kubelet/pods/47a9069f-1ffa-4c83-be8b-5e62f1d3275a/volumes" Nov 02 01:46:35.535345 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:35.535332 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="706df1bd-19cf-4ab9-9aaa-3c12c97ccf45" path="/var/lib/kubelet/pods/706df1bd-19cf-4ab9-9aaa-3c12c97ccf45/volumes" Nov 02 01:46:35.536107 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:35.536088 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb533b5f-3525-454a-8ce2-ea9246194f28" path="/var/lib/kubelet/pods/fb533b5f-3525-454a-8ce2-ea9246194f28/volumes" Nov 02 01:46:38.351286 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:38.351247 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" podUID="eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189" containerName="main" containerID="cri-o://027a348e1131304bdf6ccde156cb01bda7fa1239b9c9fad3a6022508c50d43e4" gracePeriod=45 Nov 02 01:46:38.504252 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:38.504218 4427 generic.go:356] "Generic (PLEG): container finished" podID="eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189" containerID="027a348e1131304bdf6ccde156cb01bda7fa1239b9c9fad3a6022508c50d43e4" exitCode=0 Nov 02 01:46:38.504445 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:38.504298 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" event={"ID":"eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189","Type":"ContainerDied","Data":"027a348e1131304bdf6ccde156cb01bda7fa1239b9c9fad3a6022508c50d43e4"} Nov 02 01:46:38.629260 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:38.629236 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" Nov 02 01:46:38.681214 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:38.681183 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189-dshm\" (UniqueName: \"kubernetes.io/empty-dir/eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189-dshm\") pod \"eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189\" (UID: \"eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189\") " Nov 02 01:46:38.681445 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:38.681328 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189-home\" (UniqueName: \"kubernetes.io/empty-dir/eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189-home\") pod \"eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189\" (UID: \"eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189\") " Nov 02 01:46:38.681533 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:38.681447 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189-model-cache\") pod \"eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189\" (UID: \"eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189\") " Nov 02 01:46:38.681533 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:38.681486 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189-tmp-dir\") pod \"eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189\" (UID: \"eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189\") " Nov 02 01:46:38.682094 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:38.682055 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189-home" pod "eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189" (UID: "eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:46:38.682184 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:38.682078 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189-kube-api-access-vnfmd\" (UniqueName: \"kubernetes.io/projected/eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189-kube-api-access-vnfmd\") pod \"eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189\" (UID: \"eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189\") " Nov 02 01:46:38.682251 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:38.682238 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189-tls-certs\" (UniqueName: \"kubernetes.io/secret/eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189-tls-certs\") pod \"eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189\" (UID: \"eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189\") " Nov 02 01:46:38.683657 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:38.683622 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189-model-cache" pod "eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189" (UID: "eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:46:38.683804 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:38.683711 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189-tmp-dir" pod "eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189" (UID: "eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:46:38.691097 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:38.691055 4427 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189-home\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:38.691263 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:38.691190 4427 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189-model-cache\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:38.691263 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:38.691212 4427 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189-tmp-dir\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:38.718415 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:38.713739 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189-kube-api-access-vnfmd" pod "eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189" (UID: "eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189"). InnerVolumeSpecName "kube-api-access-vnfmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:46:38.718415 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:38.713944 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189-dshm" pod "eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189" (UID: "eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:46:38.718415 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:38.715733 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189-tls-certs" pod "eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189" (UID: "eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:46:38.795325 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:38.794871 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnfmd\" (UniqueName: \"kubernetes.io/projected/eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189-kube-api-access-vnfmd\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:38.795325 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:38.794967 4427 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189-tls-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:38.795325 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:38.794983 4427 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189-dshm\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:39.512718 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:39.512689 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" Nov 02 01:46:39.512718 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:39.512703 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t" event={"ID":"eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189","Type":"ContainerDied","Data":"a2283896984978451d1ad9039bb8831847adeda065e7487a32f147af624ff6fd"} Nov 02 01:46:39.513223 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:39.512759 4427 scope.go:122] "RemoveContainer" containerID="027a348e1131304bdf6ccde156cb01bda7fa1239b9c9fad3a6022508c50d43e4" Nov 02 01:46:39.539491 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:39.539460 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t"] Nov 02 01:46:39.542756 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:39.542728 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-b9d816/simulator-w6-b9d816-kserve-57db454ff-sd28t"] Nov 02 01:46:40.139774 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:40.139742 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6"] Nov 02 01:46:40.140308 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:40.140293 4427 cpu_manager.go:414] "containerMap: removing container" podUID="47a9069f-1ffa-4c83-be8b-5e62f1d3275a" containerName="main" Nov 02 01:46:40.140405 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:40.140310 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47a9069f-1ffa-4c83-be8b-5e62f1d3275a" containerName="main" Nov 02 01:46:40.140405 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:40.140330 4427 cpu_manager.go:414] "containerMap: removing container" podUID="93282bc6-f0c6-48ce-bfac-998647223d42" containerName="revoke-keys" Nov 02 01:46:40.140405 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:40.140335 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93282bc6-f0c6-48ce-bfac-998647223d42" containerName="revoke-keys" Nov 02 01:46:40.140405 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:40.140348 4427 cpu_manager.go:414] "containerMap: removing container" podUID="eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189" containerName="main" Nov 02 01:46:40.140405 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:40.140353 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189" containerName="main" Nov 02 01:46:40.140405 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:40.140359 4427 cpu_manager.go:414] "containerMap: removing container" podUID="706df1bd-19cf-4ab9-9aaa-3c12c97ccf45" containerName="main" Nov 02 01:46:40.140405 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:40.140364 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="706df1bd-19cf-4ab9-9aaa-3c12c97ccf45" containerName="main" Nov 02 01:46:40.140405 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:40.140389 4427 cpu_manager.go:414] "containerMap: removing container" podUID="1c58e25b-d880-4738-98fc-2c7948a6153f" containerName="revoke-keys" Nov 02 01:46:40.140405 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:40.140394 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c58e25b-d880-4738-98fc-2c7948a6153f" containerName="revoke-keys" Nov 02 01:46:40.140686 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:40.140409 4427 cpu_manager.go:414] "containerMap: removing container" podUID="fb533b5f-3525-454a-8ce2-ea9246194f28" containerName="main" Nov 02 01:46:40.140686 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:40.140414 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb533b5f-3525-454a-8ce2-ea9246194f28" containerName="main" Nov 02 01:46:40.140686 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:40.140420 4427 cpu_manager.go:414] "containerMap: removing container" podUID="366c78ee-4aaf-4b7c-84b4-17028538cb1f" containerName="main" Nov 02 01:46:40.140686 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:40.140425 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="366c78ee-4aaf-4b7c-84b4-17028538cb1f" containerName="main" Nov 02 01:46:40.140686 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:40.140505 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="47a9069f-1ffa-4c83-be8b-5e62f1d3275a" containerName="main" Nov 02 01:46:40.140686 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:40.140515 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="93282bc6-f0c6-48ce-bfac-998647223d42" containerName="revoke-keys" Nov 02 01:46:40.140686 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:40.140522 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189" containerName="main" Nov 02 01:46:40.140686 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:40.140529 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="706df1bd-19cf-4ab9-9aaa-3c12c97ccf45" containerName="main" Nov 02 01:46:40.140686 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:40.140535 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c58e25b-d880-4738-98fc-2c7948a6153f" containerName="revoke-keys" Nov 02 01:46:40.140686 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:40.140541 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb533b5f-3525-454a-8ce2-ea9246194f28" containerName="main" Nov 02 01:46:40.140686 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:40.140545 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="366c78ee-4aaf-4b7c-84b4-17028538cb1f" containerName="main" Nov 02 01:46:40.145568 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:40.145550 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6" Nov 02 01:46:40.155296 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:40.154033 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-0f32d6-openshift-default-dockercfg-bh7jl\"" Nov 02 01:46:40.158457 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:40.158432 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6"] Nov 02 01:46:40.212909 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:40.212868 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/392681ce-fe14-4518-9697-efb94f82efdc-istiod-ca-cert\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6\" (UID: \"392681ce-fe14-4518-9697-efb94f82efdc\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6" Nov 02 01:46:40.213096 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:40.212919 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/392681ce-fe14-4518-9697-efb94f82efdc-workload-socket\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6\" (UID: \"392681ce-fe14-4518-9697-efb94f82efdc\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6" Nov 02 01:46:40.213096 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:40.213061 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/392681ce-fe14-4518-9697-efb94f82efdc-credential-socket\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6\" (UID: \"392681ce-fe14-4518-9697-efb94f82efdc\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6" Nov 02 01:46:40.213096 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:40.213088 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/392681ce-fe14-4518-9697-efb94f82efdc-istio-podinfo\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6\" (UID: \"392681ce-fe14-4518-9697-efb94f82efdc\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6" Nov 02 01:46:40.213247 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:40.213114 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/392681ce-fe14-4518-9697-efb94f82efdc-istio-data\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6\" (UID: \"392681ce-fe14-4518-9697-efb94f82efdc\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6" Nov 02 01:46:40.213247 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:40.213229 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wld9\" (UniqueName: \"kubernetes.io/projected/392681ce-fe14-4518-9697-efb94f82efdc-kube-api-access-4wld9\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6\" (UID: \"392681ce-fe14-4518-9697-efb94f82efdc\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6" Nov 02 01:46:40.213344 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:40.213302 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/392681ce-fe14-4518-9697-efb94f82efdc-istio-envoy\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6\" (UID: \"392681ce-fe14-4518-9697-efb94f82efdc\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6" Nov 02 01:46:40.213344 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:40.213336 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/392681ce-fe14-4518-9697-efb94f82efdc-istio-token\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6\" (UID: \"392681ce-fe14-4518-9697-efb94f82efdc\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6" Nov 02 01:46:40.213528 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:40.213510 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/392681ce-fe14-4518-9697-efb94f82efdc-workload-certs\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6\" (UID: \"392681ce-fe14-4518-9697-efb94f82efdc\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6" Nov 02 01:46:40.315626 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:40.315587 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/392681ce-fe14-4518-9697-efb94f82efdc-credential-socket\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6\" (UID: \"392681ce-fe14-4518-9697-efb94f82efdc\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6" Nov 02 01:46:40.315626 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:40.315634 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/392681ce-fe14-4518-9697-efb94f82efdc-istio-podinfo\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6\" (UID: \"392681ce-fe14-4518-9697-efb94f82efdc\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6" Nov 02 01:46:40.315988 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:40.315676 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/392681ce-fe14-4518-9697-efb94f82efdc-istio-data\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6\" (UID: \"392681ce-fe14-4518-9697-efb94f82efdc\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6" Nov 02 01:46:40.315988 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:40.315749 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wld9\" (UniqueName: \"kubernetes.io/projected/392681ce-fe14-4518-9697-efb94f82efdc-kube-api-access-4wld9\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6\" (UID: \"392681ce-fe14-4518-9697-efb94f82efdc\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6" Nov 02 01:46:40.315988 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:40.315843 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/392681ce-fe14-4518-9697-efb94f82efdc-istio-envoy\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6\" (UID: \"392681ce-fe14-4518-9697-efb94f82efdc\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6" Nov 02 01:46:40.315988 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:40.315882 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/392681ce-fe14-4518-9697-efb94f82efdc-istio-token\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6\" (UID: \"392681ce-fe14-4518-9697-efb94f82efdc\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6" Nov 02 01:46:40.315988 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:40.315932 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/392681ce-fe14-4518-9697-efb94f82efdc-workload-certs\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6\" (UID: \"392681ce-fe14-4518-9697-efb94f82efdc\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6" Nov 02 01:46:40.319425 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:40.317469 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/392681ce-fe14-4518-9697-efb94f82efdc-istiod-ca-cert\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6\" (UID: \"392681ce-fe14-4518-9697-efb94f82efdc\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6" Nov 02 01:46:40.319425 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:40.317547 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/392681ce-fe14-4518-9697-efb94f82efdc-workload-socket\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6\" (UID: \"392681ce-fe14-4518-9697-efb94f82efdc\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6" Nov 02 01:46:40.319425 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:40.317809 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/392681ce-fe14-4518-9697-efb94f82efdc-workload-certs\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6\" (UID: \"392681ce-fe14-4518-9697-efb94f82efdc\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6" Nov 02 01:46:40.319425 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:40.318414 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/392681ce-fe14-4518-9697-efb94f82efdc-workload-socket\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6\" (UID: \"392681ce-fe14-4518-9697-efb94f82efdc\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6" Nov 02 01:46:40.319425 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:40.318596 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/392681ce-fe14-4518-9697-efb94f82efdc-istio-data\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6\" (UID: \"392681ce-fe14-4518-9697-efb94f82efdc\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6" Nov 02 01:46:40.319425 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:40.318843 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/392681ce-fe14-4518-9697-efb94f82efdc-credential-socket\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6\" (UID: \"392681ce-fe14-4518-9697-efb94f82efdc\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6" Nov 02 01:46:40.320145 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:40.320109 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/392681ce-fe14-4518-9697-efb94f82efdc-istiod-ca-cert\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6\" (UID: \"392681ce-fe14-4518-9697-efb94f82efdc\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6" Nov 02 01:46:40.323920 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:40.323365 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/392681ce-fe14-4518-9697-efb94f82efdc-istio-envoy\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6\" (UID: \"392681ce-fe14-4518-9697-efb94f82efdc\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6" Nov 02 01:46:40.327520 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:40.325940 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/392681ce-fe14-4518-9697-efb94f82efdc-istio-podinfo\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6\" (UID: \"392681ce-fe14-4518-9697-efb94f82efdc\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6" Nov 02 01:46:40.331927 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:40.331896 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/392681ce-fe14-4518-9697-efb94f82efdc-istio-token\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6\" (UID: \"392681ce-fe14-4518-9697-efb94f82efdc\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6" Nov 02 01:46:40.332065 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:40.331974 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wld9\" (UniqueName: \"kubernetes.io/projected/392681ce-fe14-4518-9697-efb94f82efdc-kube-api-access-4wld9\") pod \"e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6\" (UID: \"392681ce-fe14-4518-9697-efb94f82efdc\") " pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6" Nov 02 01:46:40.462424 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:40.462321 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6" Nov 02 01:46:40.752554 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:40.752524 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6"] Nov 02 01:46:40.753524 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:46:40.753494 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod392681ce_fe14_4518_9697_efb94f82efdc.slice/crio-58ba9384fe1bb0540213ab2d2acd40726951c3cbaf6f3f8e5f304923ae091032 WatchSource:0}: Error finding container 58ba9384fe1bb0540213ab2d2acd40726951c3cbaf6f3f8e5f304923ae091032: Status 404 returned error can't find the container with id 58ba9384fe1bb0540213ab2d2acd40726951c3cbaf6f3f8e5f304923ae091032 Nov 02 01:46:40.756042 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:40.756003 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:46:40.756156 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:40.756087 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:46:40.756156 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:40.756131 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:46:41.534743 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:41.534701 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189" path="/var/lib/kubelet/pods/eea5c69d-8cda-4fa3-a0d1-dbc2ab6cf189/volumes" Nov 02 01:46:41.535101 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:41.535080 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6" event={"ID":"392681ce-fe14-4518-9697-efb94f82efdc","Type":"ContainerStarted","Data":"44ced00972441350c17d5df842a755ef8ecae0ab1a661e0b090a4e0a376f9591"} Nov 02 01:46:41.535149 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:41.535128 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6" event={"ID":"392681ce-fe14-4518-9697-efb94f82efdc","Type":"ContainerStarted","Data":"58ba9384fe1bb0540213ab2d2acd40726951c3cbaf6f3f8e5f304923ae091032"} Nov 02 01:46:41.559188 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:41.559112 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6" podStartSLOduration=1.559093853 podStartE2EDuration="1.559093853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:46:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:46:41.557946678 +0000 UTC m=+1906.178951887" watchObservedRunningTime="2026-11-02 01:46:41.559093853 +0000 UTC m=+1906.180099066" Nov 02 01:46:42.129404 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:42.119324 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b9d816-tsqmh"] Nov 02 01:46:42.129404 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:42.126092 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b9d816-tsqmh"] Nov 02 01:46:42.471457 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:42.463806 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6" Nov 02 01:46:42.475445 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:42.472494 4427 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Nov 02 01:46:42.475445 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:42.472561 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6" podUID="392681ce-fe14-4518-9697-efb94f82efdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Nov 02 01:46:43.478402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:43.475890 4427 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Nov 02 01:46:43.478402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:43.475961 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6" podUID="392681ce-fe14-4518-9697-efb94f82efdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Nov 02 01:46:43.534225 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:43.534167 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8231498-d145-430c-872a-b05cb8718fc7" path="/var/lib/kubelet/pods/b8231498-d145-430c-872a-b05cb8718fc7/volumes" Nov 02 01:46:44.484611 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:44.482692 4427 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Nov 02 01:46:44.484611 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:44.482760 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6" podUID="392681ce-fe14-4518-9697-efb94f82efdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Nov 02 01:46:45.484510 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:45.484472 4427 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Nov 02 01:46:45.484816 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:45.484786 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6" podUID="392681ce-fe14-4518-9697-efb94f82efdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Nov 02 01:46:45.998477 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:45.998438 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5687bc-ltx69"] Nov 02 01:46:46.003129 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:46.003098 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5687bc-ltx69"] Nov 02 01:46:46.464865 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:46.464827 4427 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Nov 02 01:46:46.465116 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:46.465091 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6" podUID="392681ce-fe14-4518-9697-efb94f82efdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Nov 02 01:46:47.467872 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:47.467838 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6" Nov 02 01:46:47.468324 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:47.468304 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6" Nov 02 01:46:47.469151 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:47.469130 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6" Nov 02 01:46:47.537682 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:47.537639 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f70ac873-92f8-4223-b214-9ea1d5898075" path="/var/lib/kubelet/pods/f70ac873-92f8-4223-b214-9ea1d5898075/volumes" Nov 02 01:46:49.553814 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:49.552568 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v"] Nov 02 01:46:49.554633 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:49.554599 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v" podUID="fc786b0e-e9d2-4e73-a48b-42b04f8b3eec" containerName="istio-proxy" containerID="cri-o://d9a44c2c3ab06359b52cb7b257e29f013b76aab16906d76e84a7b55e8b0898c5" gracePeriod=30 Nov 02 01:46:53.090512 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:53.090475 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2"] Nov 02 01:46:53.091350 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:53.091315 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2" podUID="2bcaee75-ac15-4b13-8f60-401ee60464ec" containerName="istio-proxy" containerID="cri-o://2e79b85cda3427b9b65f6c6554d1ef37b6df7d7b079e9b9f9fb4b7b705b61891" gracePeriod=30 Nov 02 01:46:54.813435 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:54.813275 4427 generic.go:356] "Generic (PLEG): container finished" podID="fc786b0e-e9d2-4e73-a48b-42b04f8b3eec" containerID="d9a44c2c3ab06359b52cb7b257e29f013b76aab16906d76e84a7b55e8b0898c5" exitCode=0 Nov 02 01:46:54.813435 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:54.813362 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v" event={"ID":"fc786b0e-e9d2-4e73-a48b-42b04f8b3eec","Type":"ContainerDied","Data":"d9a44c2c3ab06359b52cb7b257e29f013b76aab16906d76e84a7b55e8b0898c5"} Nov 02 01:46:54.814042 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:54.814013 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v" event={"ID":"fc786b0e-e9d2-4e73-a48b-42b04f8b3eec","Type":"ContainerDied","Data":"56ec173597fcd3971e892034c48f37046713b4c9307ac0f56c0482333fd92353"} Nov 02 01:46:54.814142 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:54.814060 4427 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56ec173597fcd3971e892034c48f37046713b4c9307ac0f56c0482333fd92353" Nov 02 01:46:54.817766 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:54.817745 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v" Nov 02 01:46:54.960426 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:54.958845 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc786b0e-e9d2-4e73-a48b-42b04f8b3eec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc786b0e-e9d2-4e73-a48b-42b04f8b3eec-credential-socket\") pod \"fc786b0e-e9d2-4e73-a48b-42b04f8b3eec\" (UID: \"fc786b0e-e9d2-4e73-a48b-42b04f8b3eec\") " Nov 02 01:46:54.963340 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:54.963283 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc786b0e-e9d2-4e73-a48b-42b04f8b3eec-credential-socket" pod "fc786b0e-e9d2-4e73-a48b-42b04f8b3eec" (UID: "fc786b0e-e9d2-4e73-a48b-42b04f8b3eec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:46:54.964551 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:54.964516 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc786b0e-e9d2-4e73-a48b-42b04f8b3eec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc786b0e-e9d2-4e73-a48b-42b04f8b3eec-istiod-ca-cert\") pod \"fc786b0e-e9d2-4e73-a48b-42b04f8b3eec\" (UID: \"fc786b0e-e9d2-4e73-a48b-42b04f8b3eec\") " Nov 02 01:46:54.964692 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:54.964585 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc786b0e-e9d2-4e73-a48b-42b04f8b3eec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc786b0e-e9d2-4e73-a48b-42b04f8b3eec-workload-socket\") pod \"fc786b0e-e9d2-4e73-a48b-42b04f8b3eec\" (UID: \"fc786b0e-e9d2-4e73-a48b-42b04f8b3eec\") " Nov 02 01:46:54.964692 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:54.964654 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc786b0e-e9d2-4e73-a48b-42b04f8b3eec-kube-api-access-xg747\" (UniqueName: \"kubernetes.io/projected/fc786b0e-e9d2-4e73-a48b-42b04f8b3eec-kube-api-access-xg747\") pod \"fc786b0e-e9d2-4e73-a48b-42b04f8b3eec\" (UID: \"fc786b0e-e9d2-4e73-a48b-42b04f8b3eec\") " Nov 02 01:46:54.964817 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:54.964737 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc786b0e-e9d2-4e73-a48b-42b04f8b3eec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc786b0e-e9d2-4e73-a48b-42b04f8b3eec-istio-envoy\") pod \"fc786b0e-e9d2-4e73-a48b-42b04f8b3eec\" (UID: \"fc786b0e-e9d2-4e73-a48b-42b04f8b3eec\") " Nov 02 01:46:54.964893 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:54.964835 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc786b0e-e9d2-4e73-a48b-42b04f8b3eec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc786b0e-e9d2-4e73-a48b-42b04f8b3eec-workload-certs\") pod \"fc786b0e-e9d2-4e73-a48b-42b04f8b3eec\" (UID: \"fc786b0e-e9d2-4e73-a48b-42b04f8b3eec\") " Nov 02 01:46:54.965019 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:54.964908 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc786b0e-e9d2-4e73-a48b-42b04f8b3eec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc786b0e-e9d2-4e73-a48b-42b04f8b3eec-istio-data\") pod \"fc786b0e-e9d2-4e73-a48b-42b04f8b3eec\" (UID: \"fc786b0e-e9d2-4e73-a48b-42b04f8b3eec\") " Nov 02 01:46:54.965019 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:54.964976 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc786b0e-e9d2-4e73-a48b-42b04f8b3eec-istio-token\" (UniqueName: \"kubernetes.io/projected/fc786b0e-e9d2-4e73-a48b-42b04f8b3eec-istio-token\") pod \"fc786b0e-e9d2-4e73-a48b-42b04f8b3eec\" (UID: \"fc786b0e-e9d2-4e73-a48b-42b04f8b3eec\") " Nov 02 01:46:54.965161 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:54.965034 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fc786b0e-e9d2-4e73-a48b-42b04f8b3eec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc786b0e-e9d2-4e73-a48b-42b04f8b3eec-istio-podinfo\") pod \"fc786b0e-e9d2-4e73-a48b-42b04f8b3eec\" (UID: \"fc786b0e-e9d2-4e73-a48b-42b04f8b3eec\") " Nov 02 01:46:54.965782 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:54.965740 4427 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc786b0e-e9d2-4e73-a48b-42b04f8b3eec-credential-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:54.968693 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:54.968662 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc786b0e-e9d2-4e73-a48b-42b04f8b3eec-istio-data" pod "fc786b0e-e9d2-4e73-a48b-42b04f8b3eec" (UID: "fc786b0e-e9d2-4e73-a48b-42b04f8b3eec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:46:54.972048 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:54.970294 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc786b0e-e9d2-4e73-a48b-42b04f8b3eec-istio-envoy" pod "fc786b0e-e9d2-4e73-a48b-42b04f8b3eec" (UID: "fc786b0e-e9d2-4e73-a48b-42b04f8b3eec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:46:54.972048 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:54.970896 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc786b0e-e9d2-4e73-a48b-42b04f8b3eec-workload-certs" pod "fc786b0e-e9d2-4e73-a48b-42b04f8b3eec" (UID: "fc786b0e-e9d2-4e73-a48b-42b04f8b3eec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:46:54.972048 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:54.971958 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc786b0e-e9d2-4e73-a48b-42b04f8b3eec-workload-socket" pod "fc786b0e-e9d2-4e73-a48b-42b04f8b3eec" (UID: "fc786b0e-e9d2-4e73-a48b-42b04f8b3eec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:46:54.972494 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:54.972446 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc786b0e-e9d2-4e73-a48b-42b04f8b3eec-istiod-ca-cert" pod "fc786b0e-e9d2-4e73-a48b-42b04f8b3eec" (UID: "fc786b0e-e9d2-4e73-a48b-42b04f8b3eec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:46:54.974679 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:54.973019 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc786b0e-e9d2-4e73-a48b-42b04f8b3eec-istio-podinfo" pod "fc786b0e-e9d2-4e73-a48b-42b04f8b3eec" (UID: "fc786b0e-e9d2-4e73-a48b-42b04f8b3eec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:46:54.988942 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:54.982997 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc786b0e-e9d2-4e73-a48b-42b04f8b3eec-istio-token" pod "fc786b0e-e9d2-4e73-a48b-42b04f8b3eec" (UID: "fc786b0e-e9d2-4e73-a48b-42b04f8b3eec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:46:54.988942 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:54.984944 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc786b0e-e9d2-4e73-a48b-42b04f8b3eec-kube-api-access-xg747" pod "fc786b0e-e9d2-4e73-a48b-42b04f8b3eec" (UID: "fc786b0e-e9d2-4e73-a48b-42b04f8b3eec"). InnerVolumeSpecName "kube-api-access-xg747". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:46:55.069659 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:55.069609 4427 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc786b0e-e9d2-4e73-a48b-42b04f8b3eec-istio-envoy\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:55.069659 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:55.069659 4427 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc786b0e-e9d2-4e73-a48b-42b04f8b3eec-workload-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:55.069930 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:55.069675 4427 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc786b0e-e9d2-4e73-a48b-42b04f8b3eec-istio-data\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:55.069930 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:55.069691 4427 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc786b0e-e9d2-4e73-a48b-42b04f8b3eec-istio-token\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:55.069930 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:55.069705 4427 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc786b0e-e9d2-4e73-a48b-42b04f8b3eec-istio-podinfo\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:55.069930 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:55.069719 4427 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc786b0e-e9d2-4e73-a48b-42b04f8b3eec-istiod-ca-cert\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:55.069930 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:55.069733 4427 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc786b0e-e9d2-4e73-a48b-42b04f8b3eec-workload-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:55.069930 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:55.069746 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg747\" (UniqueName: \"kubernetes.io/projected/fc786b0e-e9d2-4e73-a48b-42b04f8b3eec-kube-api-access-xg747\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:55.820114 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:55.820025 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v" Nov 02 01:46:55.844879 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:55.844840 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v"] Nov 02 01:46:55.851252 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:55.851222 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-b9d816-openshift-default-57986489bf-mql2v"] Nov 02 01:46:57.534106 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:57.534069 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc786b0e-e9d2-4e73-a48b-42b04f8b3eec" path="/var/lib/kubelet/pods/fc786b0e-e9d2-4e73-a48b-42b04f8b3eec/volumes" Nov 02 01:46:58.091143 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:58.091109 4427 scope.go:122] "RemoveContainer" containerID="d9a44c2c3ab06359b52cb7b257e29f013b76aab16906d76e84a7b55e8b0898c5" Nov 02 01:46:58.786864 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:58.786836 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2" Nov 02 01:46:58.834447 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:58.834412 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bcaee75-ac15-4b13-8f60-401ee60464ec-kube-api-access-6kfwl\" (UniqueName: \"kubernetes.io/projected/2bcaee75-ac15-4b13-8f60-401ee60464ec-kube-api-access-6kfwl\") pod \"2bcaee75-ac15-4b13-8f60-401ee60464ec\" (UID: \"2bcaee75-ac15-4b13-8f60-401ee60464ec\") " Nov 02 01:46:58.834654 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:58.834524 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bcaee75-ac15-4b13-8f60-401ee60464ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bcaee75-ac15-4b13-8f60-401ee60464ec-credential-socket\") pod \"2bcaee75-ac15-4b13-8f60-401ee60464ec\" (UID: \"2bcaee75-ac15-4b13-8f60-401ee60464ec\") " Nov 02 01:46:58.834654 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:58.834587 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bcaee75-ac15-4b13-8f60-401ee60464ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bcaee75-ac15-4b13-8f60-401ee60464ec-workload-certs\") pod \"2bcaee75-ac15-4b13-8f60-401ee60464ec\" (UID: \"2bcaee75-ac15-4b13-8f60-401ee60464ec\") " Nov 02 01:46:58.834654 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:58.834635 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bcaee75-ac15-4b13-8f60-401ee60464ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bcaee75-ac15-4b13-8f60-401ee60464ec-workload-socket\") pod \"2bcaee75-ac15-4b13-8f60-401ee60464ec\" (UID: \"2bcaee75-ac15-4b13-8f60-401ee60464ec\") " Nov 02 01:46:58.834836 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:58.834737 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bcaee75-ac15-4b13-8f60-401ee60464ec-istio-token\" (UniqueName: \"kubernetes.io/projected/2bcaee75-ac15-4b13-8f60-401ee60464ec-istio-token\") pod \"2bcaee75-ac15-4b13-8f60-401ee60464ec\" (UID: \"2bcaee75-ac15-4b13-8f60-401ee60464ec\") " Nov 02 01:46:58.834836 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:58.834802 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bcaee75-ac15-4b13-8f60-401ee60464ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bcaee75-ac15-4b13-8f60-401ee60464ec-istio-data\") pod \"2bcaee75-ac15-4b13-8f60-401ee60464ec\" (UID: \"2bcaee75-ac15-4b13-8f60-401ee60464ec\") " Nov 02 01:46:58.834947 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:58.834857 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2bcaee75-ac15-4b13-8f60-401ee60464ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bcaee75-ac15-4b13-8f60-401ee60464ec-istio-podinfo\") pod \"2bcaee75-ac15-4b13-8f60-401ee60464ec\" (UID: \"2bcaee75-ac15-4b13-8f60-401ee60464ec\") " Nov 02 01:46:58.834947 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:58.834859 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bcaee75-ac15-4b13-8f60-401ee60464ec-credential-socket" pod "2bcaee75-ac15-4b13-8f60-401ee60464ec" (UID: "2bcaee75-ac15-4b13-8f60-401ee60464ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:46:58.834947 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:58.834895 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bcaee75-ac15-4b13-8f60-401ee60464ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bcaee75-ac15-4b13-8f60-401ee60464ec-istiod-ca-cert\") pod \"2bcaee75-ac15-4b13-8f60-401ee60464ec\" (UID: \"2bcaee75-ac15-4b13-8f60-401ee60464ec\") " Nov 02 01:46:58.835109 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:58.834945 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bcaee75-ac15-4b13-8f60-401ee60464ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bcaee75-ac15-4b13-8f60-401ee60464ec-istio-envoy\") pod \"2bcaee75-ac15-4b13-8f60-401ee60464ec\" (UID: \"2bcaee75-ac15-4b13-8f60-401ee60464ec\") " Nov 02 01:46:58.835109 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:58.834973 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bcaee75-ac15-4b13-8f60-401ee60464ec-workload-certs" pod "2bcaee75-ac15-4b13-8f60-401ee60464ec" (UID: "2bcaee75-ac15-4b13-8f60-401ee60464ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:46:58.840068 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:58.835475 4427 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bcaee75-ac15-4b13-8f60-401ee60464ec-credential-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:58.840068 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:58.835500 4427 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bcaee75-ac15-4b13-8f60-401ee60464ec-workload-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:58.840068 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:58.836037 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bcaee75-ac15-4b13-8f60-401ee60464ec-istio-data" pod "2bcaee75-ac15-4b13-8f60-401ee60464ec" (UID: "2bcaee75-ac15-4b13-8f60-401ee60464ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:46:58.840068 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:58.838553 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bcaee75-ac15-4b13-8f60-401ee60464ec-workload-socket" pod "2bcaee75-ac15-4b13-8f60-401ee60464ec" (UID: "2bcaee75-ac15-4b13-8f60-401ee60464ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:46:58.843106 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:58.840493 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bcaee75-ac15-4b13-8f60-401ee60464ec-istiod-ca-cert" pod "2bcaee75-ac15-4b13-8f60-401ee60464ec" (UID: "2bcaee75-ac15-4b13-8f60-401ee60464ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:46:58.843106 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:58.840935 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2bcaee75-ac15-4b13-8f60-401ee60464ec-istio-podinfo" pod "2bcaee75-ac15-4b13-8f60-401ee60464ec" (UID: "2bcaee75-ac15-4b13-8f60-401ee60464ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:46:58.843106 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:58.841074 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bcaee75-ac15-4b13-8f60-401ee60464ec-kube-api-access-6kfwl" pod "2bcaee75-ac15-4b13-8f60-401ee60464ec" (UID: "2bcaee75-ac15-4b13-8f60-401ee60464ec"). InnerVolumeSpecName "kube-api-access-6kfwl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:46:58.843106 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:58.841095 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bcaee75-ac15-4b13-8f60-401ee60464ec-istio-token" pod "2bcaee75-ac15-4b13-8f60-401ee60464ec" (UID: "2bcaee75-ac15-4b13-8f60-401ee60464ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:46:58.844493 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:58.844467 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bcaee75-ac15-4b13-8f60-401ee60464ec-istio-envoy" pod "2bcaee75-ac15-4b13-8f60-401ee60464ec" (UID: "2bcaee75-ac15-4b13-8f60-401ee60464ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:46:58.844705 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:58.844682 4427 generic.go:356] "Generic (PLEG): container finished" podID="2bcaee75-ac15-4b13-8f60-401ee60464ec" containerID="2e79b85cda3427b9b65f6c6554d1ef37b6df7d7b079e9b9f9fb4b7b705b61891" exitCode=0 Nov 02 01:46:58.844822 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:58.844805 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2" event={"ID":"2bcaee75-ac15-4b13-8f60-401ee60464ec","Type":"ContainerDied","Data":"2e79b85cda3427b9b65f6c6554d1ef37b6df7d7b079e9b9f9fb4b7b705b61891"} Nov 02 01:46:58.844899 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:58.844887 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2" event={"ID":"2bcaee75-ac15-4b13-8f60-401ee60464ec","Type":"ContainerDied","Data":"b42249434a59c6c6e7df6e7cff34b2aa31dcd7aed42f2b4b2b27de5ac6185e8f"} Nov 02 01:46:58.844954 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:58.844921 4427 scope.go:122] "RemoveContainer" containerID="2e79b85cda3427b9b65f6c6554d1ef37b6df7d7b079e9b9f9fb4b7b705b61891" Nov 02 01:46:58.845086 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:58.845074 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2" Nov 02 01:46:58.884234 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:58.884208 4427 scope.go:122] "RemoveContainer" containerID="2e79b85cda3427b9b65f6c6554d1ef37b6df7d7b079e9b9f9fb4b7b705b61891" Nov 02 01:46:58.884636 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:46:58.884610 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e79b85cda3427b9b65f6c6554d1ef37b6df7d7b079e9b9f9fb4b7b705b61891\": container with ID starting with 2e79b85cda3427b9b65f6c6554d1ef37b6df7d7b079e9b9f9fb4b7b705b61891 not found: ID does not exist" containerID="2e79b85cda3427b9b65f6c6554d1ef37b6df7d7b079e9b9f9fb4b7b705b61891" Nov 02 01:46:58.884712 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:58.884645 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e79b85cda3427b9b65f6c6554d1ef37b6df7d7b079e9b9f9fb4b7b705b61891"} err="failed to get container status \"2e79b85cda3427b9b65f6c6554d1ef37b6df7d7b079e9b9f9fb4b7b705b61891\": rpc error: code = NotFound desc = could not find container \"2e79b85cda3427b9b65f6c6554d1ef37b6df7d7b079e9b9f9fb4b7b705b61891\": container with ID starting with 2e79b85cda3427b9b65f6c6554d1ef37b6df7d7b079e9b9f9fb4b7b705b61891 not found: ID does not exist" Nov 02 01:46:58.893744 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:58.893156 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2"] Nov 02 01:46:58.897090 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:58.897058 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5687bc-openshift-default-6dfd66cf9c-558z2"] Nov 02 01:46:58.936871 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:58.936836 4427 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bcaee75-ac15-4b13-8f60-401ee60464ec-istio-token\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:58.936871 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:58.936864 4427 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bcaee75-ac15-4b13-8f60-401ee60464ec-istio-data\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:58.936871 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:58.936874 4427 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bcaee75-ac15-4b13-8f60-401ee60464ec-istio-podinfo\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:58.936871 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:58.936882 4427 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bcaee75-ac15-4b13-8f60-401ee60464ec-istiod-ca-cert\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:58.937236 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:58.936891 4427 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bcaee75-ac15-4b13-8f60-401ee60464ec-istio-envoy\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:58.937236 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:58.936901 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kfwl\" (UniqueName: \"kubernetes.io/projected/2bcaee75-ac15-4b13-8f60-401ee60464ec-kube-api-access-6kfwl\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:58.937236 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:58.936910 4427 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bcaee75-ac15-4b13-8f60-401ee60464ec-workload-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:46:59.534204 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:46:59.534170 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bcaee75-ac15-4b13-8f60-401ee60464ec" path="/var/lib/kubelet/pods/2bcaee75-ac15-4b13-8f60-401ee60464ec/volumes" Nov 02 01:47:01.584934 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:47:01.584892 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:47:31.593591 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:47:31.593496 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:47:37.030223 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:47:37.030176 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bc773aa-g7wls"] Nov 02 01:47:37.032837 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:47:37.032811 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bc773aa-g7wls"] Nov 02 01:47:37.533739 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:47:37.533706 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="990a3aa9-9b89-4f58-832c-1e257cee5120" path="/var/lib/kubelet/pods/990a3aa9-9b89-4f58-832c-1e257cee5120/volumes" Nov 02 01:47:47.685519 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:47:47.685461 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0584dd92-5vd79"] Nov 02 01:47:47.686288 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:47:47.686265 4427 cpu_manager.go:414] "containerMap: removing container" podUID="fc786b0e-e9d2-4e73-a48b-42b04f8b3eec" containerName="istio-proxy" Nov 02 01:47:47.686353 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:47:47.686294 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc786b0e-e9d2-4e73-a48b-42b04f8b3eec" containerName="istio-proxy" Nov 02 01:47:47.686353 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:47:47.686334 4427 cpu_manager.go:414] "containerMap: removing container" podUID="2bcaee75-ac15-4b13-8f60-401ee60464ec" containerName="istio-proxy" Nov 02 01:47:47.686353 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:47:47.686343 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bcaee75-ac15-4b13-8f60-401ee60464ec" containerName="istio-proxy" Nov 02 01:47:47.686483 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:47:47.686473 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bcaee75-ac15-4b13-8f60-401ee60464ec" containerName="istio-proxy" Nov 02 01:47:47.686524 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:47:47.686493 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc786b0e-e9d2-4e73-a48b-42b04f8b3eec" containerName="istio-proxy" Nov 02 01:47:47.691227 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:47:47.691210 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0584dd92-5vd79" Nov 02 01:47:47.696079 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:47:47.696056 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 01:47:47.696241 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:47:47.696133 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\"" Nov 02 01:47:47.699537 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:47:47.699515 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0584dd92-5vd79"] Nov 02 01:47:47.768773 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:47:47.768730 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/656ff417-fb98-487e-9929-aa0b5c00d091-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0584dd92-5vd79\" (UID: \"656ff417-fb98-487e-9929-aa0b5c00d091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0584dd92-5vd79" Nov 02 01:47:47.768980 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:47:47.768887 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/656ff417-fb98-487e-9929-aa0b5c00d091-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0584dd92-5vd79\" (UID: \"656ff417-fb98-487e-9929-aa0b5c00d091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0584dd92-5vd79" Nov 02 01:47:47.769056 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:47:47.769038 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vsqp\" (UniqueName: \"kubernetes.io/projected/656ff417-fb98-487e-9929-aa0b5c00d091-kube-api-access-6vsqp\") pod \"maas-api-revoke-sub-0584dd92-5vd79\" (UID: \"656ff417-fb98-487e-9929-aa0b5c00d091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0584dd92-5vd79" Nov 02 01:47:47.869999 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:47:47.869964 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/656ff417-fb98-487e-9929-aa0b5c00d091-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0584dd92-5vd79\" (UID: \"656ff417-fb98-487e-9929-aa0b5c00d091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0584dd92-5vd79" Nov 02 01:47:47.870219 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:47:47.870072 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vsqp\" (UniqueName: \"kubernetes.io/projected/656ff417-fb98-487e-9929-aa0b5c00d091-kube-api-access-6vsqp\") pod \"maas-api-revoke-sub-0584dd92-5vd79\" (UID: \"656ff417-fb98-487e-9929-aa0b5c00d091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0584dd92-5vd79" Nov 02 01:47:47.870219 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:47:47.870102 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/656ff417-fb98-487e-9929-aa0b5c00d091-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0584dd92-5vd79\" (UID: \"656ff417-fb98-487e-9929-aa0b5c00d091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0584dd92-5vd79" Nov 02 01:47:47.870827 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:47:47.870802 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/656ff417-fb98-487e-9929-aa0b5c00d091-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0584dd92-5vd79\" (UID: \"656ff417-fb98-487e-9929-aa0b5c00d091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0584dd92-5vd79" Nov 02 01:47:47.872659 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:47:47.872632 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/656ff417-fb98-487e-9929-aa0b5c00d091-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0584dd92-5vd79\" (UID: \"656ff417-fb98-487e-9929-aa0b5c00d091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0584dd92-5vd79" Nov 02 01:47:47.882266 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:47:47.882234 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vsqp\" (UniqueName: \"kubernetes.io/projected/656ff417-fb98-487e-9929-aa0b5c00d091-kube-api-access-6vsqp\") pod \"maas-api-revoke-sub-0584dd92-5vd79\" (UID: \"656ff417-fb98-487e-9929-aa0b5c00d091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0584dd92-5vd79" Nov 02 01:47:48.007631 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:47:48.007544 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0584dd92-5vd79" Nov 02 01:47:48.291316 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:47:48.291284 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0584dd92-5vd79"] Nov 02 01:47:48.291694 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:47:48.291669 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod656ff417_fb98_487e_9929_aa0b5c00d091.slice/crio-fd891e1df54ee6fe79789c5518357338b8c42972eec1093b2ff8100e93c682a4 WatchSource:0}: Error finding container fd891e1df54ee6fe79789c5518357338b8c42972eec1093b2ff8100e93c682a4: Status 404 returned error can't find the container with id fd891e1df54ee6fe79789c5518357338b8c42972eec1093b2ff8100e93c682a4 Nov 02 01:47:49.178368 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:47:49.178327 4427 generic.go:356] "Generic (PLEG): container finished" podID="656ff417-fb98-487e-9929-aa0b5c00d091" containerID="59cdae7a95e0dbfec6043083207e44d95517a4f385f688d68f9315068d15dbaa" exitCode=0 Nov 02 01:47:49.178845 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:47:49.178496 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0584dd92-5vd79" event={"ID":"656ff417-fb98-487e-9929-aa0b5c00d091","Type":"ContainerDied","Data":"59cdae7a95e0dbfec6043083207e44d95517a4f385f688d68f9315068d15dbaa"} Nov 02 01:47:49.178845 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:47:49.178546 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0584dd92-5vd79" event={"ID":"656ff417-fb98-487e-9929-aa0b5c00d091","Type":"ContainerStarted","Data":"fd891e1df54ee6fe79789c5518357338b8c42972eec1093b2ff8100e93c682a4"} Nov 02 01:47:50.358476 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:47:50.358450 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0584dd92-5vd79" Nov 02 01:47:50.411416 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:47:50.411361 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/656ff417-fb98-487e-9929-aa0b5c00d091-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/656ff417-fb98-487e-9929-aa0b5c00d091-maas-api-xks-ca\") pod \"656ff417-fb98-487e-9929-aa0b5c00d091\" (UID: \"656ff417-fb98-487e-9929-aa0b5c00d091\") " Nov 02 01:47:50.411627 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:47:50.411526 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/656ff417-fb98-487e-9929-aa0b5c00d091-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/656ff417-fb98-487e-9929-aa0b5c00d091-maas-api-openshift-ca\") pod \"656ff417-fb98-487e-9929-aa0b5c00d091\" (UID: \"656ff417-fb98-487e-9929-aa0b5c00d091\") " Nov 02 01:47:50.411627 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:47:50.411593 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/656ff417-fb98-487e-9929-aa0b5c00d091-kube-api-access-6vsqp\" (UniqueName: \"kubernetes.io/projected/656ff417-fb98-487e-9929-aa0b5c00d091-kube-api-access-6vsqp\") pod \"656ff417-fb98-487e-9929-aa0b5c00d091\" (UID: \"656ff417-fb98-487e-9929-aa0b5c00d091\") " Nov 02 01:47:50.411985 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:47:50.411952 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/656ff417-fb98-487e-9929-aa0b5c00d091-maas-api-openshift-ca" pod "656ff417-fb98-487e-9929-aa0b5c00d091" (UID: "656ff417-fb98-487e-9929-aa0b5c00d091"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:47:50.412172 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:47:50.412152 4427 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/656ff417-fb98-487e-9929-aa0b5c00d091-maas-api-openshift-ca\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:47:50.413935 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:47:50.413908 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/656ff417-fb98-487e-9929-aa0b5c00d091-maas-api-xks-ca" pod "656ff417-fb98-487e-9929-aa0b5c00d091" (UID: "656ff417-fb98-487e-9929-aa0b5c00d091"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:47:50.414525 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:47:50.414500 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/656ff417-fb98-487e-9929-aa0b5c00d091-kube-api-access-6vsqp" pod "656ff417-fb98-487e-9929-aa0b5c00d091" (UID: "656ff417-fb98-487e-9929-aa0b5c00d091"). InnerVolumeSpecName "kube-api-access-6vsqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:47:50.513912 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:47:50.513825 4427 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/656ff417-fb98-487e-9929-aa0b5c00d091-maas-api-xks-ca\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:47:50.513912 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:47:50.513857 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vsqp\" (UniqueName: \"kubernetes.io/projected/656ff417-fb98-487e-9929-aa0b5c00d091-kube-api-access-6vsqp\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:47:51.192033 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:47:51.192002 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0584dd92-5vd79" Nov 02 01:47:51.192220 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:47:51.192037 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0584dd92-5vd79" event={"ID":"656ff417-fb98-487e-9929-aa0b5c00d091","Type":"ContainerDied","Data":"fd891e1df54ee6fe79789c5518357338b8c42972eec1093b2ff8100e93c682a4"} Nov 02 01:47:51.192220 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:47:51.192084 4427 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd891e1df54ee6fe79789c5518357338b8c42972eec1093b2ff8100e93c682a4" Nov 02 01:47:57.897638 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:47:57.897597 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc1a1c55-xjk8t"] Nov 02 01:47:57.898166 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:47:57.898152 4427 cpu_manager.go:414] "containerMap: removing container" podUID="656ff417-fb98-487e-9929-aa0b5c00d091" containerName="revoke-keys" Nov 02 01:47:57.898212 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:47:57.898167 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="656ff417-fb98-487e-9929-aa0b5c00d091" containerName="revoke-keys" Nov 02 01:47:57.898271 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:47:57.898262 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="656ff417-fb98-487e-9929-aa0b5c00d091" containerName="revoke-keys" Nov 02 01:47:57.901803 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:47:57.901782 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1a1c55-xjk8t" Nov 02 01:47:57.907672 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:47:57.907648 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 01:47:57.910307 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:47:57.909211 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\"" Nov 02 01:47:57.912863 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:47:57.912419 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc1a1c55-xjk8t"] Nov 02 01:47:58.025737 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:47:58.025693 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b92c9725-5685-4fe0-94b7-53225178c84f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc1a1c55-xjk8t\" (UID: \"b92c9725-5685-4fe0-94b7-53225178c84f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1a1c55-xjk8t" Nov 02 01:47:58.026109 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:47:58.026082 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9w6g\" (UniqueName: \"kubernetes.io/projected/b92c9725-5685-4fe0-94b7-53225178c84f-kube-api-access-b9w6g\") pod \"maas-api-revoke-sub-bc1a1c55-xjk8t\" (UID: \"b92c9725-5685-4fe0-94b7-53225178c84f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1a1c55-xjk8t" Nov 02 01:47:58.026725 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:47:58.026686 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b92c9725-5685-4fe0-94b7-53225178c84f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc1a1c55-xjk8t\" (UID: \"b92c9725-5685-4fe0-94b7-53225178c84f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1a1c55-xjk8t" Nov 02 01:47:58.127826 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:47:58.127783 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9w6g\" (UniqueName: \"kubernetes.io/projected/b92c9725-5685-4fe0-94b7-53225178c84f-kube-api-access-b9w6g\") pod \"maas-api-revoke-sub-bc1a1c55-xjk8t\" (UID: \"b92c9725-5685-4fe0-94b7-53225178c84f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1a1c55-xjk8t" Nov 02 01:47:58.128021 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:47:58.127863 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b92c9725-5685-4fe0-94b7-53225178c84f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc1a1c55-xjk8t\" (UID: \"b92c9725-5685-4fe0-94b7-53225178c84f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1a1c55-xjk8t" Nov 02 01:47:58.128021 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:47:58.128008 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b92c9725-5685-4fe0-94b7-53225178c84f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc1a1c55-xjk8t\" (UID: \"b92c9725-5685-4fe0-94b7-53225178c84f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1a1c55-xjk8t" Nov 02 01:47:58.128685 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:47:58.128660 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b92c9725-5685-4fe0-94b7-53225178c84f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc1a1c55-xjk8t\" (UID: \"b92c9725-5685-4fe0-94b7-53225178c84f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1a1c55-xjk8t" Nov 02 01:47:58.130410 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:47:58.130389 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b92c9725-5685-4fe0-94b7-53225178c84f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc1a1c55-xjk8t\" (UID: \"b92c9725-5685-4fe0-94b7-53225178c84f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1a1c55-xjk8t" Nov 02 01:47:58.142559 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:47:58.142536 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9w6g\" (UniqueName: \"kubernetes.io/projected/b92c9725-5685-4fe0-94b7-53225178c84f-kube-api-access-b9w6g\") pod \"maas-api-revoke-sub-bc1a1c55-xjk8t\" (UID: \"b92c9725-5685-4fe0-94b7-53225178c84f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1a1c55-xjk8t" Nov 02 01:47:58.220434 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:47:58.220403 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1a1c55-xjk8t" Nov 02 01:47:58.502697 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:47:58.502670 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc1a1c55-xjk8t"] Nov 02 01:47:58.504125 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:47:58.504094 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb92c9725_5685_4fe0_94b7_53225178c84f.slice/crio-ce759170d2f6e8a56d2a2362e6f9379472a5a518aab2dc8cc36991d755afd8a1 WatchSource:0}: Error finding container ce759170d2f6e8a56d2a2362e6f9379472a5a518aab2dc8cc36991d755afd8a1: Status 404 returned error can't find the container with id ce759170d2f6e8a56d2a2362e6f9379472a5a518aab2dc8cc36991d755afd8a1 Nov 02 01:47:59.258132 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:47:59.258095 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1a1c55-xjk8t" event={"ID":"b92c9725-5685-4fe0-94b7-53225178c84f","Type":"ContainerStarted","Data":"7735d0fe7e4c1230cae26a2eba168e1fd41adfd7fc9ee73abd16135f612016fc"} Nov 02 01:47:59.258572 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:47:59.258150 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1a1c55-xjk8t" event={"ID":"b92c9725-5685-4fe0-94b7-53225178c84f","Type":"ContainerStarted","Data":"ce759170d2f6e8a56d2a2362e6f9379472a5a518aab2dc8cc36991d755afd8a1"} Nov 02 01:47:59.279794 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:47:59.279743 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1a1c55-xjk8t" podStartSLOduration=2.279728361 podStartE2EDuration="2.279728361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:47:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:47:59.276820487 +0000 UTC m=+1983.897825699" watchObservedRunningTime="2026-11-02 01:47:59.279728361 +0000 UTC m=+1983.900733640" Nov 02 01:48:00.269517 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:00.269477 4427 generic.go:356] "Generic (PLEG): container finished" podID="b92c9725-5685-4fe0-94b7-53225178c84f" containerID="7735d0fe7e4c1230cae26a2eba168e1fd41adfd7fc9ee73abd16135f612016fc" exitCode=0 Nov 02 01:48:00.269979 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:00.269628 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1a1c55-xjk8t" event={"ID":"b92c9725-5685-4fe0-94b7-53225178c84f","Type":"ContainerDied","Data":"7735d0fe7e4c1230cae26a2eba168e1fd41adfd7fc9ee73abd16135f612016fc"} Nov 02 01:48:01.421892 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:01.421864 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1a1c55-xjk8t" Nov 02 01:48:01.578022 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:01.577934 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b92c9725-5685-4fe0-94b7-53225178c84f-kube-api-access-b9w6g\" (UniqueName: \"kubernetes.io/projected/b92c9725-5685-4fe0-94b7-53225178c84f-kube-api-access-b9w6g\") pod \"b92c9725-5685-4fe0-94b7-53225178c84f\" (UID: \"b92c9725-5685-4fe0-94b7-53225178c84f\") " Nov 02 01:48:01.578022 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:01.577990 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b92c9725-5685-4fe0-94b7-53225178c84f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b92c9725-5685-4fe0-94b7-53225178c84f-maas-api-openshift-ca\") pod \"b92c9725-5685-4fe0-94b7-53225178c84f\" (UID: \"b92c9725-5685-4fe0-94b7-53225178c84f\") " Nov 02 01:48:01.578232 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:01.578179 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b92c9725-5685-4fe0-94b7-53225178c84f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b92c9725-5685-4fe0-94b7-53225178c84f-maas-api-xks-ca\") pod \"b92c9725-5685-4fe0-94b7-53225178c84f\" (UID: \"b92c9725-5685-4fe0-94b7-53225178c84f\") " Nov 02 01:48:01.578467 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:01.578435 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b92c9725-5685-4fe0-94b7-53225178c84f-maas-api-openshift-ca" pod "b92c9725-5685-4fe0-94b7-53225178c84f" (UID: "b92c9725-5685-4fe0-94b7-53225178c84f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:48:01.578999 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:01.578975 4427 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b92c9725-5685-4fe0-94b7-53225178c84f-maas-api-openshift-ca\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:01.580527 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:01.580503 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b92c9725-5685-4fe0-94b7-53225178c84f-maas-api-xks-ca" pod "b92c9725-5685-4fe0-94b7-53225178c84f" (UID: "b92c9725-5685-4fe0-94b7-53225178c84f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:48:01.580693 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:01.580665 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b92c9725-5685-4fe0-94b7-53225178c84f-kube-api-access-b9w6g" pod "b92c9725-5685-4fe0-94b7-53225178c84f" (UID: "b92c9725-5685-4fe0-94b7-53225178c84f"). InnerVolumeSpecName "kube-api-access-b9w6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:48:01.595233 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:48:01.595195 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:48:01.679782 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:01.679749 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9w6g\" (UniqueName: \"kubernetes.io/projected/b92c9725-5685-4fe0-94b7-53225178c84f-kube-api-access-b9w6g\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:01.679782 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:01.679779 4427 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b92c9725-5685-4fe0-94b7-53225178c84f-maas-api-xks-ca\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:02.286034 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:02.285997 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1a1c55-xjk8t" Nov 02 01:48:02.286313 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:02.286029 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc1a1c55-xjk8t" event={"ID":"b92c9725-5685-4fe0-94b7-53225178c84f","Type":"ContainerDied","Data":"ce759170d2f6e8a56d2a2362e6f9379472a5a518aab2dc8cc36991d755afd8a1"} Nov 02 01:48:02.286313 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:02.286121 4427 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce759170d2f6e8a56d2a2362e6f9379472a5a518aab2dc8cc36991d755afd8a1" Nov 02 01:48:06.963672 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:06.963639 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae252552-ffm4k"] Nov 02 01:48:06.964420 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:06.964399 4427 cpu_manager.go:414] "containerMap: removing container" podUID="b92c9725-5685-4fe0-94b7-53225178c84f" containerName="revoke-keys" Nov 02 01:48:06.964420 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:06.964420 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b92c9725-5685-4fe0-94b7-53225178c84f" containerName="revoke-keys" Nov 02 01:48:06.964581 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:06.964567 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="b92c9725-5685-4fe0-94b7-53225178c84f" containerName="revoke-keys" Nov 02 01:48:06.968258 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:06.968240 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae252552-ffm4k" Nov 02 01:48:06.972941 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:06.972909 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 01:48:06.974233 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:06.974210 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\"" Nov 02 01:48:06.974417 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:06.974366 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8ae4e73-de49-4950-a87b-7c08cd7cfe7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae252552-ffm4k\" (UID: \"c8ae4e73-de49-4950-a87b-7c08cd7cfe7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae252552-ffm4k" Nov 02 01:48:06.974624 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:06.974517 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae252552-ffm4k"] Nov 02 01:48:06.974739 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:06.974723 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8ae4e73-de49-4950-a87b-7c08cd7cfe7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae252552-ffm4k\" (UID: \"c8ae4e73-de49-4950-a87b-7c08cd7cfe7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae252552-ffm4k" Nov 02 01:48:06.974806 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:06.974789 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6xw7\" (UniqueName: \"kubernetes.io/projected/c8ae4e73-de49-4950-a87b-7c08cd7cfe7f-kube-api-access-d6xw7\") pod \"maas-api-revoke-sub-ae252552-ffm4k\" (UID: \"c8ae4e73-de49-4950-a87b-7c08cd7cfe7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae252552-ffm4k" Nov 02 01:48:07.076331 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:07.076294 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8ae4e73-de49-4950-a87b-7c08cd7cfe7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae252552-ffm4k\" (UID: \"c8ae4e73-de49-4950-a87b-7c08cd7cfe7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae252552-ffm4k" Nov 02 01:48:07.076544 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:07.076350 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6xw7\" (UniqueName: \"kubernetes.io/projected/c8ae4e73-de49-4950-a87b-7c08cd7cfe7f-kube-api-access-d6xw7\") pod \"maas-api-revoke-sub-ae252552-ffm4k\" (UID: \"c8ae4e73-de49-4950-a87b-7c08cd7cfe7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae252552-ffm4k" Nov 02 01:48:07.076544 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:07.076481 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8ae4e73-de49-4950-a87b-7c08cd7cfe7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae252552-ffm4k\" (UID: \"c8ae4e73-de49-4950-a87b-7c08cd7cfe7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae252552-ffm4k" Nov 02 01:48:07.077210 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:07.077120 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8ae4e73-de49-4950-a87b-7c08cd7cfe7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae252552-ffm4k\" (UID: \"c8ae4e73-de49-4950-a87b-7c08cd7cfe7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae252552-ffm4k" Nov 02 01:48:07.078884 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:07.078863 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8ae4e73-de49-4950-a87b-7c08cd7cfe7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae252552-ffm4k\" (UID: \"c8ae4e73-de49-4950-a87b-7c08cd7cfe7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae252552-ffm4k" Nov 02 01:48:07.089556 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:07.089535 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6xw7\" (UniqueName: \"kubernetes.io/projected/c8ae4e73-de49-4950-a87b-7c08cd7cfe7f-kube-api-access-d6xw7\") pod \"maas-api-revoke-sub-ae252552-ffm4k\" (UID: \"c8ae4e73-de49-4950-a87b-7c08cd7cfe7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae252552-ffm4k" Nov 02 01:48:07.288510 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:07.288425 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae252552-ffm4k" Nov 02 01:48:07.570660 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:07.570628 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae252552-ffm4k"] Nov 02 01:48:07.572205 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:48:07.572174 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8ae4e73_de49_4950_a87b_7c08cd7cfe7f.slice/crio-424b8a87c4e3672c429c3c2ff84a4eec508d867518cc16c6d90ba25e5dec201b WatchSource:0}: Error finding container 424b8a87c4e3672c429c3c2ff84a4eec508d867518cc16c6d90ba25e5dec201b: Status 404 returned error can't find the container with id 424b8a87c4e3672c429c3c2ff84a4eec508d867518cc16c6d90ba25e5dec201b Nov 02 01:48:08.330035 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:08.329938 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae252552-ffm4k" event={"ID":"c8ae4e73-de49-4950-a87b-7c08cd7cfe7f","Type":"ContainerStarted","Data":"606a3e0ea740a9d8d6fde7f999f1e75627b2a2fb5dc2919ba91e24782f032f6c"} Nov 02 01:48:08.330035 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:08.330000 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae252552-ffm4k" event={"ID":"c8ae4e73-de49-4950-a87b-7c08cd7cfe7f","Type":"ContainerStarted","Data":"424b8a87c4e3672c429c3c2ff84a4eec508d867518cc16c6d90ba25e5dec201b"} Nov 02 01:48:08.373642 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:08.373587 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae252552-ffm4k" podStartSLOduration=2.373573171 podStartE2EDuration="2.373573171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:48:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:48:08.369302304 +0000 UTC m=+1992.990307527" watchObservedRunningTime="2026-11-02 01:48:08.373573171 +0000 UTC m=+1992.994578419" Nov 02 01:48:09.338103 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:09.338066 4427 generic.go:356] "Generic (PLEG): container finished" podID="c8ae4e73-de49-4950-a87b-7c08cd7cfe7f" containerID="606a3e0ea740a9d8d6fde7f999f1e75627b2a2fb5dc2919ba91e24782f032f6c" exitCode=0 Nov 02 01:48:09.338607 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:09.338161 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae252552-ffm4k" event={"ID":"c8ae4e73-de49-4950-a87b-7c08cd7cfe7f","Type":"ContainerDied","Data":"606a3e0ea740a9d8d6fde7f999f1e75627b2a2fb5dc2919ba91e24782f032f6c"} Nov 02 01:48:10.500087 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:10.500059 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae252552-ffm4k" Nov 02 01:48:10.520959 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:10.520918 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8ae4e73-de49-4950-a87b-7c08cd7cfe7f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8ae4e73-de49-4950-a87b-7c08cd7cfe7f-maas-api-openshift-ca\") pod \"c8ae4e73-de49-4950-a87b-7c08cd7cfe7f\" (UID: \"c8ae4e73-de49-4950-a87b-7c08cd7cfe7f\") " Nov 02 01:48:10.521146 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:10.521018 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8ae4e73-de49-4950-a87b-7c08cd7cfe7f-kube-api-access-d6xw7\" (UniqueName: \"kubernetes.io/projected/c8ae4e73-de49-4950-a87b-7c08cd7cfe7f-kube-api-access-d6xw7\") pod \"c8ae4e73-de49-4950-a87b-7c08cd7cfe7f\" (UID: \"c8ae4e73-de49-4950-a87b-7c08cd7cfe7f\") " Nov 02 01:48:10.521774 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:10.521528 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8ae4e73-de49-4950-a87b-7c08cd7cfe7f-maas-api-openshift-ca" pod "c8ae4e73-de49-4950-a87b-7c08cd7cfe7f" (UID: "c8ae4e73-de49-4950-a87b-7c08cd7cfe7f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:48:10.522253 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:10.522222 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8ae4e73-de49-4950-a87b-7c08cd7cfe7f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8ae4e73-de49-4950-a87b-7c08cd7cfe7f-maas-api-xks-ca\") pod \"c8ae4e73-de49-4950-a87b-7c08cd7cfe7f\" (UID: \"c8ae4e73-de49-4950-a87b-7c08cd7cfe7f\") " Nov 02 01:48:10.522952 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:10.522785 4427 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8ae4e73-de49-4950-a87b-7c08cd7cfe7f-maas-api-openshift-ca\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:10.524500 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:10.524471 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8ae4e73-de49-4950-a87b-7c08cd7cfe7f-kube-api-access-d6xw7" pod "c8ae4e73-de49-4950-a87b-7c08cd7cfe7f" (UID: "c8ae4e73-de49-4950-a87b-7c08cd7cfe7f"). InnerVolumeSpecName "kube-api-access-d6xw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:48:10.525247 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:10.525219 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8ae4e73-de49-4950-a87b-7c08cd7cfe7f-maas-api-xks-ca" pod "c8ae4e73-de49-4950-a87b-7c08cd7cfe7f" (UID: "c8ae4e73-de49-4950-a87b-7c08cd7cfe7f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:48:10.623667 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:10.623632 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6xw7\" (UniqueName: \"kubernetes.io/projected/c8ae4e73-de49-4950-a87b-7c08cd7cfe7f-kube-api-access-d6xw7\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:10.623667 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:10.623666 4427 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8ae4e73-de49-4950-a87b-7c08cd7cfe7f-maas-api-xks-ca\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:11.010169 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:11.010068 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-677312-6nn67"] Nov 02 01:48:11.010731 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:11.010714 4427 cpu_manager.go:414] "containerMap: removing container" podUID="c8ae4e73-de49-4950-a87b-7c08cd7cfe7f" containerName="revoke-keys" Nov 02 01:48:11.010731 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:11.010731 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8ae4e73-de49-4950-a87b-7c08cd7cfe7f" containerName="revoke-keys" Nov 02 01:48:11.010902 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:11.010818 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8ae4e73-de49-4950-a87b-7c08cd7cfe7f" containerName="revoke-keys" Nov 02 01:48:11.014571 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:11.014549 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-677312-6nn67" Nov 02 01:48:11.023457 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:11.023429 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-677312-6nn67"] Nov 02 01:48:11.032857 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:11.032823 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp2tk\" (UniqueName: \"kubernetes.io/projected/7a921f2a-a506-46b6-a464-2fe2a1dc677b-kube-api-access-tp2tk\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-677312-6nn67\" (UID: \"7a921f2a-a506-46b6-a464-2fe2a1dc677b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-677312-6nn67" Nov 02 01:48:11.033223 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:11.033201 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a921f2a-a506-46b6-a464-2fe2a1dc677b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-677312-6nn67\" (UID: \"7a921f2a-a506-46b6-a464-2fe2a1dc677b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-677312-6nn67" Nov 02 01:48:11.033570 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:11.033548 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a921f2a-a506-46b6-a464-2fe2a1dc677b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-677312-6nn67\" (UID: \"7a921f2a-a506-46b6-a464-2fe2a1dc677b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-677312-6nn67" Nov 02 01:48:11.135618 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:11.135576 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp2tk\" (UniqueName: \"kubernetes.io/projected/7a921f2a-a506-46b6-a464-2fe2a1dc677b-kube-api-access-tp2tk\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-677312-6nn67\" (UID: \"7a921f2a-a506-46b6-a464-2fe2a1dc677b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-677312-6nn67" Nov 02 01:48:11.135843 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:11.135643 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a921f2a-a506-46b6-a464-2fe2a1dc677b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-677312-6nn67\" (UID: \"7a921f2a-a506-46b6-a464-2fe2a1dc677b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-677312-6nn67" Nov 02 01:48:11.135929 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:11.135907 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a921f2a-a506-46b6-a464-2fe2a1dc677b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-677312-6nn67\" (UID: \"7a921f2a-a506-46b6-a464-2fe2a1dc677b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-677312-6nn67" Nov 02 01:48:11.140979 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:11.140857 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a921f2a-a506-46b6-a464-2fe2a1dc677b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-677312-6nn67\" (UID: \"7a921f2a-a506-46b6-a464-2fe2a1dc677b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-677312-6nn67" Nov 02 01:48:11.156672 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:11.156637 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp2tk\" (UniqueName: \"kubernetes.io/projected/7a921f2a-a506-46b6-a464-2fe2a1dc677b-kube-api-access-tp2tk\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-677312-6nn67\" (UID: \"7a921f2a-a506-46b6-a464-2fe2a1dc677b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-677312-6nn67" Nov 02 01:48:11.162440 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:11.162343 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a921f2a-a506-46b6-a464-2fe2a1dc677b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-677312-6nn67\" (UID: \"7a921f2a-a506-46b6-a464-2fe2a1dc677b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-677312-6nn67" Nov 02 01:48:11.336617 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:11.336524 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-677312-6nn67" Nov 02 01:48:11.354917 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:11.354893 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae252552-ffm4k" Nov 02 01:48:11.355071 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:11.354934 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae252552-ffm4k" event={"ID":"c8ae4e73-de49-4950-a87b-7c08cd7cfe7f","Type":"ContainerDied","Data":"424b8a87c4e3672c429c3c2ff84a4eec508d867518cc16c6d90ba25e5dec201b"} Nov 02 01:48:11.355071 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:11.355003 4427 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="424b8a87c4e3672c429c3c2ff84a4eec508d867518cc16c6d90ba25e5dec201b" Nov 02 01:48:11.622162 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:11.622133 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-677312-6nn67"] Nov 02 01:48:11.623407 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:48:11.623351 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a921f2a_a506_46b6_a464_2fe2a1dc677b.slice/crio-3775aae5095b2d8274e8dea37dc0d30c5fea1c2fbda3613b7df4cb30302d9188 WatchSource:0}: Error finding container 3775aae5095b2d8274e8dea37dc0d30c5fea1c2fbda3613b7df4cb30302d9188: Status 404 returned error can't find the container with id 3775aae5095b2d8274e8dea37dc0d30c5fea1c2fbda3613b7df4cb30302d9188 Nov 02 01:48:12.363275 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:12.363237 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-677312-6nn67" event={"ID":"7a921f2a-a506-46b6-a464-2fe2a1dc677b","Type":"ContainerStarted","Data":"4980830abe2e70cc134b74a38c9364b7698dba6eb253f7be4fff836b818aaf67"} Nov 02 01:48:12.363516 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:12.363313 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-677312-6nn67" event={"ID":"7a921f2a-a506-46b6-a464-2fe2a1dc677b","Type":"ContainerStarted","Data":"3775aae5095b2d8274e8dea37dc0d30c5fea1c2fbda3613b7df4cb30302d9188"} Nov 02 01:48:12.382941 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:12.382887 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-677312-6nn67" podStartSLOduration=2.382872604 podStartE2EDuration="2.382872604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:48:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:48:12.380296688 +0000 UTC m=+1997.001301899" watchObservedRunningTime="2026-11-02 01:48:12.382872604 +0000 UTC m=+1997.003877856" Nov 02 01:48:13.371153 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:13.371110 4427 generic.go:356] "Generic (PLEG): container finished" podID="7a921f2a-a506-46b6-a464-2fe2a1dc677b" containerID="4980830abe2e70cc134b74a38c9364b7698dba6eb253f7be4fff836b818aaf67" exitCode=0 Nov 02 01:48:13.371671 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:13.371177 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-677312-6nn67" event={"ID":"7a921f2a-a506-46b6-a464-2fe2a1dc677b","Type":"ContainerDied","Data":"4980830abe2e70cc134b74a38c9364b7698dba6eb253f7be4fff836b818aaf67"} Nov 02 01:48:14.532259 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:14.532232 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-677312-6nn67" Nov 02 01:48:14.580201 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:14.580158 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a921f2a-a506-46b6-a464-2fe2a1dc677b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a921f2a-a506-46b6-a464-2fe2a1dc677b-maas-api-openshift-ca\") pod \"7a921f2a-a506-46b6-a464-2fe2a1dc677b\" (UID: \"7a921f2a-a506-46b6-a464-2fe2a1dc677b\") " Nov 02 01:48:14.580776 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:14.580425 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a921f2a-a506-46b6-a464-2fe2a1dc677b-kube-api-access-tp2tk\" (UniqueName: \"kubernetes.io/projected/7a921f2a-a506-46b6-a464-2fe2a1dc677b-kube-api-access-tp2tk\") pod \"7a921f2a-a506-46b6-a464-2fe2a1dc677b\" (UID: \"7a921f2a-a506-46b6-a464-2fe2a1dc677b\") " Nov 02 01:48:14.580776 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:14.580487 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a921f2a-a506-46b6-a464-2fe2a1dc677b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a921f2a-a506-46b6-a464-2fe2a1dc677b-maas-api-xks-ca\") pod \"7a921f2a-a506-46b6-a464-2fe2a1dc677b\" (UID: \"7a921f2a-a506-46b6-a464-2fe2a1dc677b\") " Nov 02 01:48:14.580776 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:14.580610 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a921f2a-a506-46b6-a464-2fe2a1dc677b-maas-api-openshift-ca" pod "7a921f2a-a506-46b6-a464-2fe2a1dc677b" (UID: "7a921f2a-a506-46b6-a464-2fe2a1dc677b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:48:14.581036 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:14.581020 4427 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a921f2a-a506-46b6-a464-2fe2a1dc677b-maas-api-openshift-ca\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:14.582995 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:14.582967 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a921f2a-a506-46b6-a464-2fe2a1dc677b-maas-api-xks-ca" pod "7a921f2a-a506-46b6-a464-2fe2a1dc677b" (UID: "7a921f2a-a506-46b6-a464-2fe2a1dc677b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:48:14.583207 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:14.583184 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a921f2a-a506-46b6-a464-2fe2a1dc677b-kube-api-access-tp2tk" pod "7a921f2a-a506-46b6-a464-2fe2a1dc677b" (UID: "7a921f2a-a506-46b6-a464-2fe2a1dc677b"). InnerVolumeSpecName "kube-api-access-tp2tk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:48:14.681924 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:14.681835 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tp2tk\" (UniqueName: \"kubernetes.io/projected/7a921f2a-a506-46b6-a464-2fe2a1dc677b-kube-api-access-tp2tk\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:14.681924 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:14.681866 4427 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a921f2a-a506-46b6-a464-2fe2a1dc677b-maas-api-xks-ca\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:15.383702 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:15.383667 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-677312-6nn67" Nov 02 01:48:15.383702 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:15.383691 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-677312-6nn67" event={"ID":"7a921f2a-a506-46b6-a464-2fe2a1dc677b","Type":"ContainerDied","Data":"3775aae5095b2d8274e8dea37dc0d30c5fea1c2fbda3613b7df4cb30302d9188"} Nov 02 01:48:15.383917 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:15.383747 4427 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3775aae5095b2d8274e8dea37dc0d30c5fea1c2fbda3613b7df4cb30302d9188" Nov 02 01:48:17.316495 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:17.304648 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75ad5e26-rhcrt"] Nov 02 01:48:17.316495 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:17.306097 4427 cpu_manager.go:414] "containerMap: removing container" podUID="7a921f2a-a506-46b6-a464-2fe2a1dc677b" containerName="revoke-keys" Nov 02 01:48:17.316495 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:17.306121 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a921f2a-a506-46b6-a464-2fe2a1dc677b" containerName="revoke-keys" Nov 02 01:48:17.316495 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:17.306474 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a921f2a-a506-46b6-a464-2fe2a1dc677b" containerName="revoke-keys" Nov 02 01:48:17.316495 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:17.311722 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ad5e26-rhcrt" Nov 02 01:48:17.318868 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:17.318835 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 01:48:17.319018 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:17.318972 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\"" Nov 02 01:48:17.319740 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:17.319715 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75ad5e26-rhcrt"] Nov 02 01:48:17.425337 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:17.425296 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b8eef49-4f84-4080-9006-27b77583d1ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75ad5e26-rhcrt\" (UID: \"7b8eef49-4f84-4080-9006-27b77583d1ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ad5e26-rhcrt" Nov 02 01:48:17.425592 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:17.425367 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7xk5\" (UniqueName: \"kubernetes.io/projected/7b8eef49-4f84-4080-9006-27b77583d1ea-kube-api-access-n7xk5\") pod \"maas-api-revoke-sub-75ad5e26-rhcrt\" (UID: \"7b8eef49-4f84-4080-9006-27b77583d1ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ad5e26-rhcrt" Nov 02 01:48:17.425592 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:17.425481 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b8eef49-4f84-4080-9006-27b77583d1ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75ad5e26-rhcrt\" (UID: \"7b8eef49-4f84-4080-9006-27b77583d1ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ad5e26-rhcrt" Nov 02 01:48:17.527195 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:17.527162 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b8eef49-4f84-4080-9006-27b77583d1ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75ad5e26-rhcrt\" (UID: \"7b8eef49-4f84-4080-9006-27b77583d1ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ad5e26-rhcrt" Nov 02 01:48:17.527431 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:17.527243 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7xk5\" (UniqueName: \"kubernetes.io/projected/7b8eef49-4f84-4080-9006-27b77583d1ea-kube-api-access-n7xk5\") pod \"maas-api-revoke-sub-75ad5e26-rhcrt\" (UID: \"7b8eef49-4f84-4080-9006-27b77583d1ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ad5e26-rhcrt" Nov 02 01:48:17.527431 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:17.527270 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b8eef49-4f84-4080-9006-27b77583d1ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75ad5e26-rhcrt\" (UID: \"7b8eef49-4f84-4080-9006-27b77583d1ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ad5e26-rhcrt" Nov 02 01:48:17.528154 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:17.528124 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b8eef49-4f84-4080-9006-27b77583d1ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75ad5e26-rhcrt\" (UID: \"7b8eef49-4f84-4080-9006-27b77583d1ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ad5e26-rhcrt" Nov 02 01:48:17.532721 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:17.532694 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b8eef49-4f84-4080-9006-27b77583d1ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75ad5e26-rhcrt\" (UID: \"7b8eef49-4f84-4080-9006-27b77583d1ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ad5e26-rhcrt" Nov 02 01:48:17.539848 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:17.539820 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7xk5\" (UniqueName: \"kubernetes.io/projected/7b8eef49-4f84-4080-9006-27b77583d1ea-kube-api-access-n7xk5\") pod \"maas-api-revoke-sub-75ad5e26-rhcrt\" (UID: \"7b8eef49-4f84-4080-9006-27b77583d1ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ad5e26-rhcrt" Nov 02 01:48:17.637786 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:17.637758 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ad5e26-rhcrt" Nov 02 01:48:17.922480 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:17.922448 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75ad5e26-rhcrt"] Nov 02 01:48:17.923140 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:48:17.923114 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b8eef49_4f84_4080_9006_27b77583d1ea.slice/crio-a49ee2a46643faf97c293ac77f3cab57ceb2cc923759e08ef9c493b1e62eb6fb WatchSource:0}: Error finding container a49ee2a46643faf97c293ac77f3cab57ceb2cc923759e08ef9c493b1e62eb6fb: Status 404 returned error can't find the container with id a49ee2a46643faf97c293ac77f3cab57ceb2cc923759e08ef9c493b1e62eb6fb Nov 02 01:48:18.404975 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:18.404939 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ad5e26-rhcrt" event={"ID":"7b8eef49-4f84-4080-9006-27b77583d1ea","Type":"ContainerStarted","Data":"4f1fa6c5e7219500344e7487e6d1487a3e45ed2f2572005be2769b61f3c18a01"} Nov 02 01:48:18.405448 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:18.404994 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ad5e26-rhcrt" event={"ID":"7b8eef49-4f84-4080-9006-27b77583d1ea","Type":"ContainerStarted","Data":"a49ee2a46643faf97c293ac77f3cab57ceb2cc923759e08ef9c493b1e62eb6fb"} Nov 02 01:48:18.426344 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:18.426290 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ad5e26-rhcrt" podStartSLOduration=1.426277508 podStartE2EDuration="1.426277508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:48:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:48:18.422681758 +0000 UTC m=+2003.043686967" watchObservedRunningTime="2026-11-02 01:48:18.426277508 +0000 UTC m=+2003.047282718" Nov 02 01:48:19.413390 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:19.413345 4427 generic.go:356] "Generic (PLEG): container finished" podID="7b8eef49-4f84-4080-9006-27b77583d1ea" containerID="4f1fa6c5e7219500344e7487e6d1487a3e45ed2f2572005be2769b61f3c18a01" exitCode=0 Nov 02 01:48:19.413795 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:19.413438 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ad5e26-rhcrt" event={"ID":"7b8eef49-4f84-4080-9006-27b77583d1ea","Type":"ContainerDied","Data":"4f1fa6c5e7219500344e7487e6d1487a3e45ed2f2572005be2769b61f3c18a01"} Nov 02 01:48:20.604570 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:20.604545 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ad5e26-rhcrt" Nov 02 01:48:20.688242 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:20.688166 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b8eef49-4f84-4080-9006-27b77583d1ea-kube-api-access-n7xk5\" (UniqueName: \"kubernetes.io/projected/7b8eef49-4f84-4080-9006-27b77583d1ea-kube-api-access-n7xk5\") pod \"7b8eef49-4f84-4080-9006-27b77583d1ea\" (UID: \"7b8eef49-4f84-4080-9006-27b77583d1ea\") " Nov 02 01:48:20.688434 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:20.688298 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b8eef49-4f84-4080-9006-27b77583d1ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b8eef49-4f84-4080-9006-27b77583d1ea-maas-api-openshift-ca\") pod \"7b8eef49-4f84-4080-9006-27b77583d1ea\" (UID: \"7b8eef49-4f84-4080-9006-27b77583d1ea\") " Nov 02 01:48:20.688434 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:20.688348 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b8eef49-4f84-4080-9006-27b77583d1ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b8eef49-4f84-4080-9006-27b77583d1ea-maas-api-xks-ca\") pod \"7b8eef49-4f84-4080-9006-27b77583d1ea\" (UID: \"7b8eef49-4f84-4080-9006-27b77583d1ea\") " Nov 02 01:48:20.688687 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:20.688660 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b8eef49-4f84-4080-9006-27b77583d1ea-maas-api-openshift-ca" pod "7b8eef49-4f84-4080-9006-27b77583d1ea" (UID: "7b8eef49-4f84-4080-9006-27b77583d1ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:48:20.690474 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:20.690441 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b8eef49-4f84-4080-9006-27b77583d1ea-kube-api-access-n7xk5" pod "7b8eef49-4f84-4080-9006-27b77583d1ea" (UID: "7b8eef49-4f84-4080-9006-27b77583d1ea"). InnerVolumeSpecName "kube-api-access-n7xk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:48:20.690575 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:20.690533 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b8eef49-4f84-4080-9006-27b77583d1ea-maas-api-xks-ca" pod "7b8eef49-4f84-4080-9006-27b77583d1ea" (UID: "7b8eef49-4f84-4080-9006-27b77583d1ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:48:20.790391 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:20.790325 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7xk5\" (UniqueName: \"kubernetes.io/projected/7b8eef49-4f84-4080-9006-27b77583d1ea-kube-api-access-n7xk5\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:20.790584 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:20.790418 4427 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b8eef49-4f84-4080-9006-27b77583d1ea-maas-api-openshift-ca\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:20.790584 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:20.790431 4427 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b8eef49-4f84-4080-9006-27b77583d1ea-maas-api-xks-ca\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:21.121642 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:21.121607 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-677312-54c7456ddc-fvmf6"] Nov 02 01:48:21.137314 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:21.137268 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-677312-6fb455f55f-q8f9v"] Nov 02 01:48:21.467583 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:21.467468 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ad5e26-rhcrt" event={"ID":"7b8eef49-4f84-4080-9006-27b77583d1ea","Type":"ContainerDied","Data":"a49ee2a46643faf97c293ac77f3cab57ceb2cc923759e08ef9c493b1e62eb6fb"} Nov 02 01:48:21.467837 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:21.467815 4427 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a49ee2a46643faf97c293ac77f3cab57ceb2cc923759e08ef9c493b1e62eb6fb" Nov 02 01:48:21.468064 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:21.468016 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ad5e26-rhcrt" Nov 02 01:48:22.974812 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:22.974777 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-677312-6nn67"] Nov 02 01:48:22.979666 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:22.979629 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-677312-6nn67"] Nov 02 01:48:23.535114 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:23.535077 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a921f2a-a506-46b6-a464-2fe2a1dc677b" path="/var/lib/kubelet/pods/7a921f2a-a506-46b6-a464-2fe2a1dc677b/volumes" Nov 02 01:48:26.122038 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:26.121982 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-677312-54c7456ddc-fvmf6" podUID="d133c468-9c0c-4d15-8b0e-9292b27944f6" containerName="payload-processing" containerID="cri-o://c4616ecde2911c4639b4be90570a5e27c0bf897d2ba101d2fef378d4cd0e9be5" gracePeriod=25 Nov 02 01:48:26.138354 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:26.138301 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-677312-6fb455f55f-q8f9v" podUID="317a9ada-6f39-4dee-8945-9cc2dd1c18eb" containerName="payload-pre-processing" containerID="cri-o://95c6d74cf33ff7289e109f010c4a398e8733ad63840ceff059fc44061b75999c" gracePeriod=25 Nov 02 01:48:31.061838 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:31.061798 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3c4ea5-tr2p7"] Nov 02 01:48:31.063291 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:31.063265 4427 cpu_manager.go:414] "containerMap: removing container" podUID="7b8eef49-4f84-4080-9006-27b77583d1ea" containerName="revoke-keys" Nov 02 01:48:31.063453 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:31.063439 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b8eef49-4f84-4080-9006-27b77583d1ea" containerName="revoke-keys" Nov 02 01:48:31.063768 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:31.063755 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b8eef49-4f84-4080-9006-27b77583d1ea" containerName="revoke-keys" Nov 02 01:48:31.074738 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:31.074685 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3c4ea5-tr2p7" Nov 02 01:48:31.097442 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:31.083195 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\"" Nov 02 01:48:31.097442 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:31.086721 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht"] Nov 02 01:48:31.097442 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:31.091562 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3c4ea5-tr2p7"] Nov 02 01:48:31.097442 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:31.091771 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht" podUID="ce4186ac-472d-4f1a-999f-7a95cffd6a05" containerName="istio-proxy" containerID="cri-o://7170332b1c843f9156b8d3c9f65c0cabfcd185cd0971a000e5e5cd68e1fb084a" gracePeriod=30 Nov 02 01:48:31.098512 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:31.086803 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 01:48:31.298260 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:31.298216 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eedfc823-180c-43b1-9b43-73e34b260b29-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-3c4ea5-tr2p7\" (UID: \"eedfc823-180c-43b1-9b43-73e34b260b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3c4ea5-tr2p7" Nov 02 01:48:31.298519 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:31.298498 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw67z\" (UniqueName: \"kubernetes.io/projected/eedfc823-180c-43b1-9b43-73e34b260b29-kube-api-access-rw67z\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-3c4ea5-tr2p7\" (UID: \"eedfc823-180c-43b1-9b43-73e34b260b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3c4ea5-tr2p7" Nov 02 01:48:31.298588 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:31.298539 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eedfc823-180c-43b1-9b43-73e34b260b29-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-3c4ea5-tr2p7\" (UID: \"eedfc823-180c-43b1-9b43-73e34b260b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3c4ea5-tr2p7" Nov 02 01:48:31.399921 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:31.399877 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw67z\" (UniqueName: \"kubernetes.io/projected/eedfc823-180c-43b1-9b43-73e34b260b29-kube-api-access-rw67z\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-3c4ea5-tr2p7\" (UID: \"eedfc823-180c-43b1-9b43-73e34b260b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3c4ea5-tr2p7" Nov 02 01:48:31.399921 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:31.399927 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eedfc823-180c-43b1-9b43-73e34b260b29-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-3c4ea5-tr2p7\" (UID: \"eedfc823-180c-43b1-9b43-73e34b260b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3c4ea5-tr2p7" Nov 02 01:48:31.400188 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:31.399985 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eedfc823-180c-43b1-9b43-73e34b260b29-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-3c4ea5-tr2p7\" (UID: \"eedfc823-180c-43b1-9b43-73e34b260b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3c4ea5-tr2p7" Nov 02 01:48:31.400685 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:31.400656 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eedfc823-180c-43b1-9b43-73e34b260b29-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-3c4ea5-tr2p7\" (UID: \"eedfc823-180c-43b1-9b43-73e34b260b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3c4ea5-tr2p7" Nov 02 01:48:31.402758 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:31.402732 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eedfc823-180c-43b1-9b43-73e34b260b29-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-3c4ea5-tr2p7\" (UID: \"eedfc823-180c-43b1-9b43-73e34b260b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3c4ea5-tr2p7" Nov 02 01:48:31.413976 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:31.413952 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw67z\" (UniqueName: \"kubernetes.io/projected/eedfc823-180c-43b1-9b43-73e34b260b29-kube-api-access-rw67z\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-3c4ea5-tr2p7\" (UID: \"eedfc823-180c-43b1-9b43-73e34b260b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3c4ea5-tr2p7" Nov 02 01:48:31.501418 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:31.501350 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-677312-6fb455f55f-q8f9v" podUID="317a9ada-6f39-4dee-8945-9cc2dd1c18eb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 01:48:31.501627 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:31.501524 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-677312-54c7456ddc-fvmf6" podUID="d133c468-9c0c-4d15-8b0e-9292b27944f6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 01:48:31.594598 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:48:31.594545 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:48:31.710630 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:31.710543 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3c4ea5-tr2p7" Nov 02 01:48:31.997988 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:48:31.997957 4427 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeedfc823_180c_43b1_9b43_73e34b260b29.slice/crio-a4501d53235e08b4b2d45015cd4b38167efda81ad520a3939669ae3470540b89\": RecentStats: unable to find data in memory cache]" Nov 02 01:48:32.000401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:32.000347 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3c4ea5-tr2p7"] Nov 02 01:48:32.539957 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:32.539854 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3c4ea5-tr2p7" event={"ID":"eedfc823-180c-43b1-9b43-73e34b260b29","Type":"ContainerStarted","Data":"6c07049a2de0f237500a61672ed0d680cfe09741a4ffe3d10d9376903b18c318"} Nov 02 01:48:32.540383 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:32.539962 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3c4ea5-tr2p7" event={"ID":"eedfc823-180c-43b1-9b43-73e34b260b29","Type":"ContainerStarted","Data":"a4501d53235e08b4b2d45015cd4b38167efda81ad520a3939669ae3470540b89"} Nov 02 01:48:32.558728 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:32.558674 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3c4ea5-tr2p7" podStartSLOduration=1.55865746 podStartE2EDuration="1.55865746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:48:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:48:32.558487484 +0000 UTC m=+2017.179492696" watchObservedRunningTime="2026-11-02 01:48:32.55865746 +0000 UTC m=+2017.179662672" Nov 02 01:48:33.547899 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:33.547865 4427 generic.go:356] "Generic (PLEG): container finished" podID="eedfc823-180c-43b1-9b43-73e34b260b29" containerID="6c07049a2de0f237500a61672ed0d680cfe09741a4ffe3d10d9376903b18c318" exitCode=0 Nov 02 01:48:33.548321 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:33.547910 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3c4ea5-tr2p7" event={"ID":"eedfc823-180c-43b1-9b43-73e34b260b29","Type":"ContainerDied","Data":"6c07049a2de0f237500a61672ed0d680cfe09741a4ffe3d10d9376903b18c318"} Nov 02 01:48:34.709899 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:34.709873 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3c4ea5-tr2p7" Nov 02 01:48:34.846838 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:34.846750 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eedfc823-180c-43b1-9b43-73e34b260b29-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eedfc823-180c-43b1-9b43-73e34b260b29-maas-api-openshift-ca\") pod \"eedfc823-180c-43b1-9b43-73e34b260b29\" (UID: \"eedfc823-180c-43b1-9b43-73e34b260b29\") " Nov 02 01:48:34.847019 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:34.846893 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eedfc823-180c-43b1-9b43-73e34b260b29-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eedfc823-180c-43b1-9b43-73e34b260b29-maas-api-xks-ca\") pod \"eedfc823-180c-43b1-9b43-73e34b260b29\" (UID: \"eedfc823-180c-43b1-9b43-73e34b260b29\") " Nov 02 01:48:34.847019 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:34.846995 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eedfc823-180c-43b1-9b43-73e34b260b29-kube-api-access-rw67z\" (UniqueName: \"kubernetes.io/projected/eedfc823-180c-43b1-9b43-73e34b260b29-kube-api-access-rw67z\") pod \"eedfc823-180c-43b1-9b43-73e34b260b29\" (UID: \"eedfc823-180c-43b1-9b43-73e34b260b29\") " Nov 02 01:48:34.847166 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:34.847137 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eedfc823-180c-43b1-9b43-73e34b260b29-maas-api-openshift-ca" pod "eedfc823-180c-43b1-9b43-73e34b260b29" (UID: "eedfc823-180c-43b1-9b43-73e34b260b29"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:48:34.847597 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:34.847571 4427 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eedfc823-180c-43b1-9b43-73e34b260b29-maas-api-openshift-ca\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:34.849143 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:34.849120 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eedfc823-180c-43b1-9b43-73e34b260b29-maas-api-xks-ca" pod "eedfc823-180c-43b1-9b43-73e34b260b29" (UID: "eedfc823-180c-43b1-9b43-73e34b260b29"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:48:34.849461 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:34.849439 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eedfc823-180c-43b1-9b43-73e34b260b29-kube-api-access-rw67z" pod "eedfc823-180c-43b1-9b43-73e34b260b29" (UID: "eedfc823-180c-43b1-9b43-73e34b260b29"). InnerVolumeSpecName "kube-api-access-rw67z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:48:34.949052 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:34.949021 4427 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eedfc823-180c-43b1-9b43-73e34b260b29-maas-api-xks-ca\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:34.949052 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:34.949050 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw67z\" (UniqueName: \"kubernetes.io/projected/eedfc823-180c-43b1-9b43-73e34b260b29-kube-api-access-rw67z\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:35.562432 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:35.562404 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3c4ea5-tr2p7" Nov 02 01:48:35.562432 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:35.562412 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3c4ea5-tr2p7" event={"ID":"eedfc823-180c-43b1-9b43-73e34b260b29","Type":"ContainerDied","Data":"a4501d53235e08b4b2d45015cd4b38167efda81ad520a3939669ae3470540b89"} Nov 02 01:48:35.562666 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:35.562475 4427 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4501d53235e08b4b2d45015cd4b38167efda81ad520a3939669ae3470540b89" Nov 02 01:48:36.364797 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:36.364739 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht" Nov 02 01:48:36.469067 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:36.469028 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ce4186ac-472d-4f1a-999f-7a95cffd6a05-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce4186ac-472d-4f1a-999f-7a95cffd6a05-istio-podinfo\") pod \"ce4186ac-472d-4f1a-999f-7a95cffd6a05\" (UID: \"ce4186ac-472d-4f1a-999f-7a95cffd6a05\") " Nov 02 01:48:36.469236 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:36.469129 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce4186ac-472d-4f1a-999f-7a95cffd6a05-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce4186ac-472d-4f1a-999f-7a95cffd6a05-workload-certs\") pod \"ce4186ac-472d-4f1a-999f-7a95cffd6a05\" (UID: \"ce4186ac-472d-4f1a-999f-7a95cffd6a05\") " Nov 02 01:48:36.469236 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:36.469182 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce4186ac-472d-4f1a-999f-7a95cffd6a05-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce4186ac-472d-4f1a-999f-7a95cffd6a05-workload-socket\") pod \"ce4186ac-472d-4f1a-999f-7a95cffd6a05\" (UID: \"ce4186ac-472d-4f1a-999f-7a95cffd6a05\") " Nov 02 01:48:36.469236 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:36.469216 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce4186ac-472d-4f1a-999f-7a95cffd6a05-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce4186ac-472d-4f1a-999f-7a95cffd6a05-credential-socket\") pod \"ce4186ac-472d-4f1a-999f-7a95cffd6a05\" (UID: \"ce4186ac-472d-4f1a-999f-7a95cffd6a05\") " Nov 02 01:48:36.469348 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:36.469306 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce4186ac-472d-4f1a-999f-7a95cffd6a05-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce4186ac-472d-4f1a-999f-7a95cffd6a05-istio-envoy\") pod \"ce4186ac-472d-4f1a-999f-7a95cffd6a05\" (UID: \"ce4186ac-472d-4f1a-999f-7a95cffd6a05\") " Nov 02 01:48:36.469436 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:36.469342 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce4186ac-472d-4f1a-999f-7a95cffd6a05-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce4186ac-472d-4f1a-999f-7a95cffd6a05-istiod-ca-cert\") pod \"ce4186ac-472d-4f1a-999f-7a95cffd6a05\" (UID: \"ce4186ac-472d-4f1a-999f-7a95cffd6a05\") " Nov 02 01:48:36.469505 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:36.469435 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce4186ac-472d-4f1a-999f-7a95cffd6a05-istio-token\" (UniqueName: \"kubernetes.io/projected/ce4186ac-472d-4f1a-999f-7a95cffd6a05-istio-token\") pod \"ce4186ac-472d-4f1a-999f-7a95cffd6a05\" (UID: \"ce4186ac-472d-4f1a-999f-7a95cffd6a05\") " Nov 02 01:48:36.469560 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:36.469505 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce4186ac-472d-4f1a-999f-7a95cffd6a05-kube-api-access-ndrps\" (UniqueName: \"kubernetes.io/projected/ce4186ac-472d-4f1a-999f-7a95cffd6a05-kube-api-access-ndrps\") pod \"ce4186ac-472d-4f1a-999f-7a95cffd6a05\" (UID: \"ce4186ac-472d-4f1a-999f-7a95cffd6a05\") " Nov 02 01:48:36.469560 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:36.469531 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce4186ac-472d-4f1a-999f-7a95cffd6a05-workload-certs" pod "ce4186ac-472d-4f1a-999f-7a95cffd6a05" (UID: "ce4186ac-472d-4f1a-999f-7a95cffd6a05"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:48:36.469664 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:36.469572 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce4186ac-472d-4f1a-999f-7a95cffd6a05-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce4186ac-472d-4f1a-999f-7a95cffd6a05-istio-data\") pod \"ce4186ac-472d-4f1a-999f-7a95cffd6a05\" (UID: \"ce4186ac-472d-4f1a-999f-7a95cffd6a05\") " Nov 02 01:48:36.469664 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:36.469615 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce4186ac-472d-4f1a-999f-7a95cffd6a05-credential-socket" pod "ce4186ac-472d-4f1a-999f-7a95cffd6a05" (UID: "ce4186ac-472d-4f1a-999f-7a95cffd6a05"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:48:36.469664 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:36.469640 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce4186ac-472d-4f1a-999f-7a95cffd6a05-workload-socket" pod "ce4186ac-472d-4f1a-999f-7a95cffd6a05" (UID: "ce4186ac-472d-4f1a-999f-7a95cffd6a05"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:48:36.469891 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:36.469871 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce4186ac-472d-4f1a-999f-7a95cffd6a05-istio-data" pod "ce4186ac-472d-4f1a-999f-7a95cffd6a05" (UID: "ce4186ac-472d-4f1a-999f-7a95cffd6a05"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:48:36.470228 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:36.470039 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce4186ac-472d-4f1a-999f-7a95cffd6a05-istiod-ca-cert" pod "ce4186ac-472d-4f1a-999f-7a95cffd6a05" (UID: "ce4186ac-472d-4f1a-999f-7a95cffd6a05"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:48:36.470312 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:36.470267 4427 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce4186ac-472d-4f1a-999f-7a95cffd6a05-istio-data\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:36.470312 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:36.470290 4427 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce4186ac-472d-4f1a-999f-7a95cffd6a05-workload-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:36.470312 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:36.470304 4427 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce4186ac-472d-4f1a-999f-7a95cffd6a05-workload-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:36.470471 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:36.470319 4427 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce4186ac-472d-4f1a-999f-7a95cffd6a05-credential-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:36.470471 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:36.470332 4427 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce4186ac-472d-4f1a-999f-7a95cffd6a05-istiod-ca-cert\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:36.471712 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:36.471675 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce4186ac-472d-4f1a-999f-7a95cffd6a05-istio-envoy" pod "ce4186ac-472d-4f1a-999f-7a95cffd6a05" (UID: "ce4186ac-472d-4f1a-999f-7a95cffd6a05"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:48:36.471712 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:36.471684 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce4186ac-472d-4f1a-999f-7a95cffd6a05-istio-podinfo" pod "ce4186ac-472d-4f1a-999f-7a95cffd6a05" (UID: "ce4186ac-472d-4f1a-999f-7a95cffd6a05"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:48:36.472275 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:36.472253 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce4186ac-472d-4f1a-999f-7a95cffd6a05-istio-token" pod "ce4186ac-472d-4f1a-999f-7a95cffd6a05" (UID: "ce4186ac-472d-4f1a-999f-7a95cffd6a05"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:48:36.472351 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:36.472267 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce4186ac-472d-4f1a-999f-7a95cffd6a05-kube-api-access-ndrps" pod "ce4186ac-472d-4f1a-999f-7a95cffd6a05" (UID: "ce4186ac-472d-4f1a-999f-7a95cffd6a05"). InnerVolumeSpecName "kube-api-access-ndrps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:48:36.570567 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:36.570528 4427 generic.go:356] "Generic (PLEG): container finished" podID="ce4186ac-472d-4f1a-999f-7a95cffd6a05" containerID="7170332b1c843f9156b8d3c9f65c0cabfcd185cd0971a000e5e5cd68e1fb084a" exitCode=0 Nov 02 01:48:36.570783 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:36.570581 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht" Nov 02 01:48:36.570783 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:36.570608 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht" event={"ID":"ce4186ac-472d-4f1a-999f-7a95cffd6a05","Type":"ContainerDied","Data":"7170332b1c843f9156b8d3c9f65c0cabfcd185cd0971a000e5e5cd68e1fb084a"} Nov 02 01:48:36.570783 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:36.570670 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht" event={"ID":"ce4186ac-472d-4f1a-999f-7a95cffd6a05","Type":"ContainerDied","Data":"efb248484b9a39c9a78f6a19c2389f61d9ea87241581348a50f9fd5fc7328822"} Nov 02 01:48:36.570783 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:36.570692 4427 scope.go:122] "RemoveContainer" containerID="7170332b1c843f9156b8d3c9f65c0cabfcd185cd0971a000e5e5cd68e1fb084a" Nov 02 01:48:36.571465 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:36.571440 4427 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce4186ac-472d-4f1a-999f-7a95cffd6a05-istio-envoy\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:36.571465 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:36.571463 4427 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce4186ac-472d-4f1a-999f-7a95cffd6a05-istio-token\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:36.571600 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:36.571478 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndrps\" (UniqueName: \"kubernetes.io/projected/ce4186ac-472d-4f1a-999f-7a95cffd6a05-kube-api-access-ndrps\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:36.571600 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:36.571487 4427 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce4186ac-472d-4f1a-999f-7a95cffd6a05-istio-podinfo\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:36.582634 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:36.582615 4427 scope.go:122] "RemoveContainer" containerID="7170332b1c843f9156b8d3c9f65c0cabfcd185cd0971a000e5e5cd68e1fb084a" Nov 02 01:48:36.582945 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:48:36.582924 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7170332b1c843f9156b8d3c9f65c0cabfcd185cd0971a000e5e5cd68e1fb084a\": container with ID starting with 7170332b1c843f9156b8d3c9f65c0cabfcd185cd0971a000e5e5cd68e1fb084a not found: ID does not exist" containerID="7170332b1c843f9156b8d3c9f65c0cabfcd185cd0971a000e5e5cd68e1fb084a" Nov 02 01:48:36.583012 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:36.582954 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7170332b1c843f9156b8d3c9f65c0cabfcd185cd0971a000e5e5cd68e1fb084a"} err="failed to get container status \"7170332b1c843f9156b8d3c9f65c0cabfcd185cd0971a000e5e5cd68e1fb084a\": rpc error: code = NotFound desc = could not find container \"7170332b1c843f9156b8d3c9f65c0cabfcd185cd0971a000e5e5cd68e1fb084a\": container with ID starting with 7170332b1c843f9156b8d3c9f65c0cabfcd185cd0971a000e5e5cd68e1fb084a not found: ID does not exist" Nov 02 01:48:36.599353 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:36.599324 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht"] Nov 02 01:48:36.606866 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:36.606840 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-677312-openshift-default-67d66447bf-tgtht"] Nov 02 01:48:37.533309 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:37.533276 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce4186ac-472d-4f1a-999f-7a95cffd6a05" path="/var/lib/kubelet/pods/ce4186ac-472d-4f1a-999f-7a95cffd6a05/volumes" Nov 02 01:48:38.226233 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:38.226195 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-eaec8c-zj7tw"] Nov 02 01:48:38.227068 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:38.227042 4427 cpu_manager.go:414] "containerMap: removing container" podUID="eedfc823-180c-43b1-9b43-73e34b260b29" containerName="revoke-keys" Nov 02 01:48:38.227068 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:38.227067 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eedfc823-180c-43b1-9b43-73e34b260b29" containerName="revoke-keys" Nov 02 01:48:38.227275 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:38.227105 4427 cpu_manager.go:414] "containerMap: removing container" podUID="ce4186ac-472d-4f1a-999f-7a95cffd6a05" containerName="istio-proxy" Nov 02 01:48:38.227275 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:38.227113 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce4186ac-472d-4f1a-999f-7a95cffd6a05" containerName="istio-proxy" Nov 02 01:48:38.227275 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:38.227267 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce4186ac-472d-4f1a-999f-7a95cffd6a05" containerName="istio-proxy" Nov 02 01:48:38.227472 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:38.227290 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="eedfc823-180c-43b1-9b43-73e34b260b29" containerName="revoke-keys" Nov 02 01:48:38.233539 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:38.233515 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-eaec8c-zj7tw" Nov 02 01:48:38.240712 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:38.240682 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-eaec8c-zj7tw"] Nov 02 01:48:38.241025 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:38.241003 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 01:48:38.242197 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:38.241676 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\"" Nov 02 01:48:38.290883 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:38.290844 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrkds\" (UniqueName: \"kubernetes.io/projected/af28d18e-199e-4c22-bb6d-9b83e7e0473a-kube-api-access-wrkds\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-eaec8c-zj7tw\" (UID: \"af28d18e-199e-4c22-bb6d-9b83e7e0473a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-eaec8c-zj7tw" Nov 02 01:48:38.291090 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:38.291060 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af28d18e-199e-4c22-bb6d-9b83e7e0473a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-eaec8c-zj7tw\" (UID: \"af28d18e-199e-4c22-bb6d-9b83e7e0473a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-eaec8c-zj7tw" Nov 02 01:48:38.291173 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:38.291156 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af28d18e-199e-4c22-bb6d-9b83e7e0473a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-eaec8c-zj7tw\" (UID: \"af28d18e-199e-4c22-bb6d-9b83e7e0473a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-eaec8c-zj7tw" Nov 02 01:48:38.392512 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:38.392474 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af28d18e-199e-4c22-bb6d-9b83e7e0473a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-eaec8c-zj7tw\" (UID: \"af28d18e-199e-4c22-bb6d-9b83e7e0473a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-eaec8c-zj7tw" Nov 02 01:48:38.392704 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:38.392545 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af28d18e-199e-4c22-bb6d-9b83e7e0473a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-eaec8c-zj7tw\" (UID: \"af28d18e-199e-4c22-bb6d-9b83e7e0473a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-eaec8c-zj7tw" Nov 02 01:48:38.392704 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:38.392627 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrkds\" (UniqueName: \"kubernetes.io/projected/af28d18e-199e-4c22-bb6d-9b83e7e0473a-kube-api-access-wrkds\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-eaec8c-zj7tw\" (UID: \"af28d18e-199e-4c22-bb6d-9b83e7e0473a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-eaec8c-zj7tw" Nov 02 01:48:38.393265 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:38.393235 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af28d18e-199e-4c22-bb6d-9b83e7e0473a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-eaec8c-zj7tw\" (UID: \"af28d18e-199e-4c22-bb6d-9b83e7e0473a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-eaec8c-zj7tw" Nov 02 01:48:38.395238 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:38.395212 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af28d18e-199e-4c22-bb6d-9b83e7e0473a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-eaec8c-zj7tw\" (UID: \"af28d18e-199e-4c22-bb6d-9b83e7e0473a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-eaec8c-zj7tw" Nov 02 01:48:38.405694 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:38.405659 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrkds\" (UniqueName: \"kubernetes.io/projected/af28d18e-199e-4c22-bb6d-9b83e7e0473a-kube-api-access-wrkds\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-eaec8c-zj7tw\" (UID: \"af28d18e-199e-4c22-bb6d-9b83e7e0473a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-eaec8c-zj7tw" Nov 02 01:48:38.565829 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:38.565738 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-eaec8c-zj7tw" Nov 02 01:48:38.850865 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:38.850838 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-eaec8c-zj7tw"] Nov 02 01:48:38.852447 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:48:38.852420 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf28d18e_199e_4c22_bb6d_9b83e7e0473a.slice/crio-3c2e8c6a5de6b4f00b96635e97964fad6953840a92d2dbb0b7f809a797346393 WatchSource:0}: Error finding container 3c2e8c6a5de6b4f00b96635e97964fad6953840a92d2dbb0b7f809a797346393: Status 404 returned error can't find the container with id 3c2e8c6a5de6b4f00b96635e97964fad6953840a92d2dbb0b7f809a797346393 Nov 02 01:48:39.596744 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:39.596638 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-eaec8c-zj7tw" event={"ID":"af28d18e-199e-4c22-bb6d-9b83e7e0473a","Type":"ContainerStarted","Data":"4b1a44c1996812e92d6d5b5fcbfaef6f393cc6fe8aad3b7cad7f4c3f50bea340"} Nov 02 01:48:39.596744 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:39.596705 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-eaec8c-zj7tw" event={"ID":"af28d18e-199e-4c22-bb6d-9b83e7e0473a","Type":"ContainerStarted","Data":"3c2e8c6a5de6b4f00b96635e97964fad6953840a92d2dbb0b7f809a797346393"} Nov 02 01:48:39.616072 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:39.616022 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-eaec8c-zj7tw" podStartSLOduration=1.616007891 podStartE2EDuration="1.616007891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:48:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:48:39.61353185 +0000 UTC m=+2024.234537085" watchObservedRunningTime="2026-11-02 01:48:39.616007891 +0000 UTC m=+2024.237013144" Nov 02 01:48:40.605273 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:40.605239 4427 generic.go:356] "Generic (PLEG): container finished" podID="af28d18e-199e-4c22-bb6d-9b83e7e0473a" containerID="4b1a44c1996812e92d6d5b5fcbfaef6f393cc6fe8aad3b7cad7f4c3f50bea340" exitCode=0 Nov 02 01:48:40.605700 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:40.605333 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-eaec8c-zj7tw" event={"ID":"af28d18e-199e-4c22-bb6d-9b83e7e0473a","Type":"ContainerDied","Data":"4b1a44c1996812e92d6d5b5fcbfaef6f393cc6fe8aad3b7cad7f4c3f50bea340"} Nov 02 01:48:41.517005 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:41.516950 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-677312-54c7456ddc-fvmf6" podUID="d133c468-9c0c-4d15-8b0e-9292b27944f6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 01:48:41.521509 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:41.521469 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-677312-6fb455f55f-q8f9v" podUID="317a9ada-6f39-4dee-8945-9cc2dd1c18eb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 01:48:41.755561 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:41.755516 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-0e06f2-64fc8f586c-8sw9n"] Nov 02 01:48:41.779170 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:41.778790 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e06f2-855bdc6447-dxbs5"] Nov 02 01:48:41.841216 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:41.841167 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-eaec8c-zj7tw" Nov 02 01:48:41.930037 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:41.930006 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3c4ea5-tr2p7"] Nov 02 01:48:41.936598 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:41.936566 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3c4ea5-tr2p7"] Nov 02 01:48:41.947795 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:41.947762 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af28d18e-199e-4c22-bb6d-9b83e7e0473a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af28d18e-199e-4c22-bb6d-9b83e7e0473a-maas-api-openshift-ca\") pod \"af28d18e-199e-4c22-bb6d-9b83e7e0473a\" (UID: \"af28d18e-199e-4c22-bb6d-9b83e7e0473a\") " Nov 02 01:48:41.948575 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:41.948551 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af28d18e-199e-4c22-bb6d-9b83e7e0473a-kube-api-access-wrkds\" (UniqueName: \"kubernetes.io/projected/af28d18e-199e-4c22-bb6d-9b83e7e0473a-kube-api-access-wrkds\") pod \"af28d18e-199e-4c22-bb6d-9b83e7e0473a\" (UID: \"af28d18e-199e-4c22-bb6d-9b83e7e0473a\") " Nov 02 01:48:41.949080 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:41.949060 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af28d18e-199e-4c22-bb6d-9b83e7e0473a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af28d18e-199e-4c22-bb6d-9b83e7e0473a-maas-api-xks-ca\") pod \"af28d18e-199e-4c22-bb6d-9b83e7e0473a\" (UID: \"af28d18e-199e-4c22-bb6d-9b83e7e0473a\") " Nov 02 01:48:41.949543 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:41.948363 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af28d18e-199e-4c22-bb6d-9b83e7e0473a-maas-api-openshift-ca" pod "af28d18e-199e-4c22-bb6d-9b83e7e0473a" (UID: "af28d18e-199e-4c22-bb6d-9b83e7e0473a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:48:41.950027 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:41.950001 4427 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af28d18e-199e-4c22-bb6d-9b83e7e0473a-maas-api-openshift-ca\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:41.964428 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:41.953172 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af28d18e-199e-4c22-bb6d-9b83e7e0473a-kube-api-access-wrkds" pod "af28d18e-199e-4c22-bb6d-9b83e7e0473a" (UID: "af28d18e-199e-4c22-bb6d-9b83e7e0473a"). InnerVolumeSpecName "kube-api-access-wrkds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:48:41.964428 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:41.953416 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af28d18e-199e-4c22-bb6d-9b83e7e0473a-maas-api-xks-ca" pod "af28d18e-199e-4c22-bb6d-9b83e7e0473a" (UID: "af28d18e-199e-4c22-bb6d-9b83e7e0473a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:48:42.057609 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:42.057512 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrkds\" (UniqueName: \"kubernetes.io/projected/af28d18e-199e-4c22-bb6d-9b83e7e0473a-kube-api-access-wrkds\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:42.057609 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:42.057556 4427 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af28d18e-199e-4c22-bb6d-9b83e7e0473a-maas-api-xks-ca\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:42.625712 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:42.625681 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-eaec8c-zj7tw" Nov 02 01:48:42.625712 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:42.625697 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-eaec8c-zj7tw" event={"ID":"af28d18e-199e-4c22-bb6d-9b83e7e0473a","Type":"ContainerDied","Data":"3c2e8c6a5de6b4f00b96635e97964fad6953840a92d2dbb0b7f809a797346393"} Nov 02 01:48:42.625944 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:42.625754 4427 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c2e8c6a5de6b4f00b96635e97964fad6953840a92d2dbb0b7f809a797346393" Nov 02 01:48:43.534700 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:43.534666 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eedfc823-180c-43b1-9b43-73e34b260b29" path="/var/lib/kubelet/pods/eedfc823-180c-43b1-9b43-73e34b260b29/volumes" Nov 02 01:48:46.519899 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:46.518509 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-677312-54c7456ddc-fvmf6" Nov 02 01:48:46.609142 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:46.609113 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-677312-6fb455f55f-q8f9v" Nov 02 01:48:46.620354 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:46.620328 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d133c468-9c0c-4d15-8b0e-9292b27944f6-kube-api-access-j9ltm\" (UniqueName: \"kubernetes.io/projected/d133c468-9c0c-4d15-8b0e-9292b27944f6-kube-api-access-j9ltm\") pod \"d133c468-9c0c-4d15-8b0e-9292b27944f6\" (UID: \"d133c468-9c0c-4d15-8b0e-9292b27944f6\") " Nov 02 01:48:46.620508 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:46.620472 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d133c468-9c0c-4d15-8b0e-9292b27944f6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d133c468-9c0c-4d15-8b0e-9292b27944f6-plugins-config-volume\") pod \"d133c468-9c0c-4d15-8b0e-9292b27944f6\" (UID: \"d133c468-9c0c-4d15-8b0e-9292b27944f6\") " Nov 02 01:48:46.620851 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:46.620826 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d133c468-9c0c-4d15-8b0e-9292b27944f6-plugins-config-volume" pod "d133c468-9c0c-4d15-8b0e-9292b27944f6" (UID: "d133c468-9c0c-4d15-8b0e-9292b27944f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:48:46.621060 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:46.621032 4427 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d133c468-9c0c-4d15-8b0e-9292b27944f6-plugins-config-volume\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:46.622790 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:46.622766 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d133c468-9c0c-4d15-8b0e-9292b27944f6-kube-api-access-j9ltm" pod "d133c468-9c0c-4d15-8b0e-9292b27944f6" (UID: "d133c468-9c0c-4d15-8b0e-9292b27944f6"). InnerVolumeSpecName "kube-api-access-j9ltm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:48:46.662856 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:46.662765 4427 generic.go:356] "Generic (PLEG): container finished" podID="317a9ada-6f39-4dee-8945-9cc2dd1c18eb" containerID="95c6d74cf33ff7289e109f010c4a398e8733ad63840ceff059fc44061b75999c" exitCode=0 Nov 02 01:48:46.663044 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:46.662878 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-677312-6fb455f55f-q8f9v" Nov 02 01:48:46.663044 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:46.662912 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-677312-6fb455f55f-q8f9v" event={"ID":"317a9ada-6f39-4dee-8945-9cc2dd1c18eb","Type":"ContainerDied","Data":"95c6d74cf33ff7289e109f010c4a398e8733ad63840ceff059fc44061b75999c"} Nov 02 01:48:46.663044 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:46.662984 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-677312-6fb455f55f-q8f9v" event={"ID":"317a9ada-6f39-4dee-8945-9cc2dd1c18eb","Type":"ContainerDied","Data":"d5601a6b2046bd179407adb5cb7df8c0fb1195b6d0ec6abc65874185955ad661"} Nov 02 01:48:46.663044 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:46.663016 4427 scope.go:122] "RemoveContainer" containerID="95c6d74cf33ff7289e109f010c4a398e8733ad63840ceff059fc44061b75999c" Nov 02 01:48:46.664814 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:46.664785 4427 generic.go:356] "Generic (PLEG): container finished" podID="d133c468-9c0c-4d15-8b0e-9292b27944f6" containerID="c4616ecde2911c4639b4be90570a5e27c0bf897d2ba101d2fef378d4cd0e9be5" exitCode=0 Nov 02 01:48:46.664814 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:46.664801 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-677312-54c7456ddc-fvmf6" event={"ID":"d133c468-9c0c-4d15-8b0e-9292b27944f6","Type":"ContainerDied","Data":"c4616ecde2911c4639b4be90570a5e27c0bf897d2ba101d2fef378d4cd0e9be5"} Nov 02 01:48:46.665010 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:46.664850 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-677312-54c7456ddc-fvmf6" event={"ID":"d133c468-9c0c-4d15-8b0e-9292b27944f6","Type":"ContainerDied","Data":"5a776d891dfb88d01c8c4f293a8e369e67b4d8f16954c929666cf6ab7489970e"} Nov 02 01:48:46.665010 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:46.664889 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-677312-54c7456ddc-fvmf6" Nov 02 01:48:46.708207 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:46.708180 4427 scope.go:122] "RemoveContainer" containerID="95c6d74cf33ff7289e109f010c4a398e8733ad63840ceff059fc44061b75999c" Nov 02 01:48:46.708898 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:48:46.708867 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95c6d74cf33ff7289e109f010c4a398e8733ad63840ceff059fc44061b75999c\": container with ID starting with 95c6d74cf33ff7289e109f010c4a398e8733ad63840ceff059fc44061b75999c not found: ID does not exist" containerID="95c6d74cf33ff7289e109f010c4a398e8733ad63840ceff059fc44061b75999c" Nov 02 01:48:46.709023 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:46.708909 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95c6d74cf33ff7289e109f010c4a398e8733ad63840ceff059fc44061b75999c"} err="failed to get container status \"95c6d74cf33ff7289e109f010c4a398e8733ad63840ceff059fc44061b75999c\": rpc error: code = NotFound desc = could not find container \"95c6d74cf33ff7289e109f010c4a398e8733ad63840ceff059fc44061b75999c\": container with ID starting with 95c6d74cf33ff7289e109f010c4a398e8733ad63840ceff059fc44061b75999c not found: ID does not exist" Nov 02 01:48:46.709023 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:46.708926 4427 scope.go:122] "RemoveContainer" containerID="c4616ecde2911c4639b4be90570a5e27c0bf897d2ba101d2fef378d4cd0e9be5" Nov 02 01:48:46.718220 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:46.717752 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-677312-54c7456ddc-fvmf6"] Nov 02 01:48:46.724421 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:46.721131 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-677312-54c7456ddc-fvmf6"] Nov 02 01:48:46.724421 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:46.721505 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/317a9ada-6f39-4dee-8945-9cc2dd1c18eb-kube-api-access-m46cj\" (UniqueName: \"kubernetes.io/projected/317a9ada-6f39-4dee-8945-9cc2dd1c18eb-kube-api-access-m46cj\") pod \"317a9ada-6f39-4dee-8945-9cc2dd1c18eb\" (UID: \"317a9ada-6f39-4dee-8945-9cc2dd1c18eb\") " Nov 02 01:48:46.724421 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:46.721628 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/317a9ada-6f39-4dee-8945-9cc2dd1c18eb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/317a9ada-6f39-4dee-8945-9cc2dd1c18eb-plugins-config-volume\") pod \"317a9ada-6f39-4dee-8945-9cc2dd1c18eb\" (UID: \"317a9ada-6f39-4dee-8945-9cc2dd1c18eb\") " Nov 02 01:48:46.724421 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:46.721939 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9ltm\" (UniqueName: \"kubernetes.io/projected/d133c468-9c0c-4d15-8b0e-9292b27944f6-kube-api-access-j9ltm\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:46.724421 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:46.722327 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/317a9ada-6f39-4dee-8945-9cc2dd1c18eb-plugins-config-volume" pod "317a9ada-6f39-4dee-8945-9cc2dd1c18eb" (UID: "317a9ada-6f39-4dee-8945-9cc2dd1c18eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:48:46.724421 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:46.723718 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/317a9ada-6f39-4dee-8945-9cc2dd1c18eb-kube-api-access-m46cj" pod "317a9ada-6f39-4dee-8945-9cc2dd1c18eb" (UID: "317a9ada-6f39-4dee-8945-9cc2dd1c18eb"). InnerVolumeSpecName "kube-api-access-m46cj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:48:46.726533 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:46.726510 4427 scope.go:122] "RemoveContainer" containerID="c4616ecde2911c4639b4be90570a5e27c0bf897d2ba101d2fef378d4cd0e9be5" Nov 02 01:48:46.726843 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:48:46.726823 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4616ecde2911c4639b4be90570a5e27c0bf897d2ba101d2fef378d4cd0e9be5\": container with ID starting with c4616ecde2911c4639b4be90570a5e27c0bf897d2ba101d2fef378d4cd0e9be5 not found: ID does not exist" containerID="c4616ecde2911c4639b4be90570a5e27c0bf897d2ba101d2fef378d4cd0e9be5" Nov 02 01:48:46.726915 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:46.726852 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4616ecde2911c4639b4be90570a5e27c0bf897d2ba101d2fef378d4cd0e9be5"} err="failed to get container status \"c4616ecde2911c4639b4be90570a5e27c0bf897d2ba101d2fef378d4cd0e9be5\": rpc error: code = NotFound desc = could not find container \"c4616ecde2911c4639b4be90570a5e27c0bf897d2ba101d2fef378d4cd0e9be5\": container with ID starting with c4616ecde2911c4639b4be90570a5e27c0bf897d2ba101d2fef378d4cd0e9be5 not found: ID does not exist" Nov 02 01:48:46.755986 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:46.755952 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e06f2-64fc8f586c-8sw9n" podUID="79fb94dd-4426-43ed-af58-cfa762e77b13" containerName="payload-processing" containerID="cri-o://d7c4ee8f3187db030a01706818be0cad0f93975643eef0c147bd0ba2ba6e98dc" gracePeriod=25 Nov 02 01:48:46.780629 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:46.780567 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e06f2-855bdc6447-dxbs5" podUID="aceec71b-ffcb-47e7-964e-84bc8405f2de" containerName="payload-pre-processing" containerID="cri-o://9f85df429f76202a4141848bceefc109dd8f3ee26d5e6b2333455bf8f40c719f" gracePeriod=25 Nov 02 01:48:46.822868 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:46.822835 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m46cj\" (UniqueName: \"kubernetes.io/projected/317a9ada-6f39-4dee-8945-9cc2dd1c18eb-kube-api-access-m46cj\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:46.822868 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:46.822865 4427 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/317a9ada-6f39-4dee-8945-9cc2dd1c18eb-plugins-config-volume\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:47.000498 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:47.000467 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-677312-6fb455f55f-q8f9v"] Nov 02 01:48:47.011501 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:47.011462 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-677312-6fb455f55f-q8f9v"] Nov 02 01:48:47.534647 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:47.534604 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="317a9ada-6f39-4dee-8945-9cc2dd1c18eb" path="/var/lib/kubelet/pods/317a9ada-6f39-4dee-8945-9cc2dd1c18eb/volumes" Nov 02 01:48:47.535088 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:47.535012 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d133c468-9c0c-4d15-8b0e-9292b27944f6" path="/var/lib/kubelet/pods/d133c468-9c0c-4d15-8b0e-9292b27944f6/volumes" Nov 02 01:48:51.902276 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:51.902234 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-eaec8c-zj7tw"] Nov 02 01:48:51.907603 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:51.907571 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-eaec8c-zj7tw"] Nov 02 01:48:53.100740 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:53.100699 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e06f2-855bdc6447-dxbs5" podUID="aceec71b-ffcb-47e7-964e-84bc8405f2de" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 01:48:53.101133 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:53.100709 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e06f2-64fc8f586c-8sw9n" podUID="79fb94dd-4426-43ed-af58-cfa762e77b13" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 01:48:53.533695 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:53.533662 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af28d18e-199e-4c22-bb6d-9b83e7e0473a" path="/var/lib/kubelet/pods/af28d18e-199e-4c22-bb6d-9b83e7e0473a/volumes" Nov 02 01:48:54.078483 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:54.078301 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp"] Nov 02 01:48:54.078692 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:54.078571 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp" podUID="8f9b0669-291d-43fa-9511-d74948b03684" containerName="istio-proxy" containerID="cri-o://f3bbaed735d3a00f7bdd6497f8dc6c54770bb6cbf4923ddb3073775664a538eb" gracePeriod=30 Nov 02 01:48:58.173184 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:58.173149 4427 scope.go:122] "RemoveContainer" containerID="e3ed2d7b4272def2ab3a40a0560adbdd93645673103e0cafadc8d3cf736bfd02" Nov 02 01:48:58.183470 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:58.183445 4427 scope.go:122] "RemoveContainer" containerID="8123d1bd02c0f65ae153c107c4879b5c63fe02dce6ac0dce9db7e98d011da67e" Nov 02 01:48:59.356848 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:59.356784 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp" Nov 02 01:48:59.398005 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:59.397971 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f9b0669-291d-43fa-9511-d74948b03684-istio-token\" (UniqueName: \"kubernetes.io/projected/8f9b0669-291d-43fa-9511-d74948b03684-istio-token\") pod \"8f9b0669-291d-43fa-9511-d74948b03684\" (UID: \"8f9b0669-291d-43fa-9511-d74948b03684\") " Nov 02 01:48:59.398197 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:59.398063 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f9b0669-291d-43fa-9511-d74948b03684-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f9b0669-291d-43fa-9511-d74948b03684-istio-data\") pod \"8f9b0669-291d-43fa-9511-d74948b03684\" (UID: \"8f9b0669-291d-43fa-9511-d74948b03684\") " Nov 02 01:48:59.398197 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:59.398129 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f9b0669-291d-43fa-9511-d74948b03684-kube-api-access-qg7v2\" (UniqueName: \"kubernetes.io/projected/8f9b0669-291d-43fa-9511-d74948b03684-kube-api-access-qg7v2\") pod \"8f9b0669-291d-43fa-9511-d74948b03684\" (UID: \"8f9b0669-291d-43fa-9511-d74948b03684\") " Nov 02 01:48:59.398197 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:59.398148 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f9b0669-291d-43fa-9511-d74948b03684-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f9b0669-291d-43fa-9511-d74948b03684-istiod-ca-cert\") pod \"8f9b0669-291d-43fa-9511-d74948b03684\" (UID: \"8f9b0669-291d-43fa-9511-d74948b03684\") " Nov 02 01:48:59.398197 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:59.398195 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f9b0669-291d-43fa-9511-d74948b03684-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f9b0669-291d-43fa-9511-d74948b03684-credential-socket\") pod \"8f9b0669-291d-43fa-9511-d74948b03684\" (UID: \"8f9b0669-291d-43fa-9511-d74948b03684\") " Nov 02 01:48:59.398466 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:59.398245 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f9b0669-291d-43fa-9511-d74948b03684-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f9b0669-291d-43fa-9511-d74948b03684-workload-certs\") pod \"8f9b0669-291d-43fa-9511-d74948b03684\" (UID: \"8f9b0669-291d-43fa-9511-d74948b03684\") " Nov 02 01:48:59.398466 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:59.398301 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8f9b0669-291d-43fa-9511-d74948b03684-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f9b0669-291d-43fa-9511-d74948b03684-istio-podinfo\") pod \"8f9b0669-291d-43fa-9511-d74948b03684\" (UID: \"8f9b0669-291d-43fa-9511-d74948b03684\") " Nov 02 01:48:59.398466 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:59.398408 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f9b0669-291d-43fa-9511-d74948b03684-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f9b0669-291d-43fa-9511-d74948b03684-workload-socket\") pod \"8f9b0669-291d-43fa-9511-d74948b03684\" (UID: \"8f9b0669-291d-43fa-9511-d74948b03684\") " Nov 02 01:48:59.398466 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:59.398449 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f9b0669-291d-43fa-9511-d74948b03684-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f9b0669-291d-43fa-9511-d74948b03684-istio-envoy\") pod \"8f9b0669-291d-43fa-9511-d74948b03684\" (UID: \"8f9b0669-291d-43fa-9511-d74948b03684\") " Nov 02 01:48:59.398680 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:59.398459 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f9b0669-291d-43fa-9511-d74948b03684-istio-data" pod "8f9b0669-291d-43fa-9511-d74948b03684" (UID: "8f9b0669-291d-43fa-9511-d74948b03684"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:48:59.398680 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:59.398546 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f9b0669-291d-43fa-9511-d74948b03684-credential-socket" pod "8f9b0669-291d-43fa-9511-d74948b03684" (UID: "8f9b0669-291d-43fa-9511-d74948b03684"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:48:59.398680 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:59.398569 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f9b0669-291d-43fa-9511-d74948b03684-workload-certs" pod "8f9b0669-291d-43fa-9511-d74948b03684" (UID: "8f9b0669-291d-43fa-9511-d74948b03684"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:48:59.398830 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:59.398695 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f9b0669-291d-43fa-9511-d74948b03684-workload-socket" pod "8f9b0669-291d-43fa-9511-d74948b03684" (UID: "8f9b0669-291d-43fa-9511-d74948b03684"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:48:59.398977 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:59.398949 4427 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f9b0669-291d-43fa-9511-d74948b03684-workload-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:59.399045 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:59.398990 4427 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f9b0669-291d-43fa-9511-d74948b03684-istio-data\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:59.399045 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:59.399004 4427 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f9b0669-291d-43fa-9511-d74948b03684-credential-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:59.399045 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:59.399019 4427 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f9b0669-291d-43fa-9511-d74948b03684-workload-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:59.399045 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:59.398962 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f9b0669-291d-43fa-9511-d74948b03684-istiod-ca-cert" pod "8f9b0669-291d-43fa-9511-d74948b03684" (UID: "8f9b0669-291d-43fa-9511-d74948b03684"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:48:59.400524 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:59.400496 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f9b0669-291d-43fa-9511-d74948b03684-istio-token" pod "8f9b0669-291d-43fa-9511-d74948b03684" (UID: "8f9b0669-291d-43fa-9511-d74948b03684"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:48:59.400670 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:59.400650 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f9b0669-291d-43fa-9511-d74948b03684-kube-api-access-qg7v2" pod "8f9b0669-291d-43fa-9511-d74948b03684" (UID: "8f9b0669-291d-43fa-9511-d74948b03684"). InnerVolumeSpecName "kube-api-access-qg7v2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:48:59.400800 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:59.400778 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f9b0669-291d-43fa-9511-d74948b03684-istio-podinfo" pod "8f9b0669-291d-43fa-9511-d74948b03684" (UID: "8f9b0669-291d-43fa-9511-d74948b03684"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:48:59.400922 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:59.400895 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f9b0669-291d-43fa-9511-d74948b03684-istio-envoy" pod "8f9b0669-291d-43fa-9511-d74948b03684" (UID: "8f9b0669-291d-43fa-9511-d74948b03684"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:48:59.477854 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:59.477817 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22"] Nov 02 01:48:59.478419 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:59.478331 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22" podUID="f31dc5a1-ea54-4858-ac4c-2b990192ee4c" containerName="istio-proxy" containerID="cri-o://b9a3277d2256fe4b6a16984a340fd19b7a21f7eb6724b4912989a4d1bba47422" gracePeriod=30 Nov 02 01:48:59.499881 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:59.499844 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qg7v2\" (UniqueName: \"kubernetes.io/projected/8f9b0669-291d-43fa-9511-d74948b03684-kube-api-access-qg7v2\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:59.499881 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:59.499885 4427 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f9b0669-291d-43fa-9511-d74948b03684-istiod-ca-cert\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:59.500117 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:59.499896 4427 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f9b0669-291d-43fa-9511-d74948b03684-istio-podinfo\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:59.500117 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:59.499905 4427 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f9b0669-291d-43fa-9511-d74948b03684-istio-envoy\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:59.500117 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:59.499914 4427 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f9b0669-291d-43fa-9511-d74948b03684-istio-token\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:48:59.776316 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:59.776274 4427 generic.go:356] "Generic (PLEG): container finished" podID="8f9b0669-291d-43fa-9511-d74948b03684" containerID="f3bbaed735d3a00f7bdd6497f8dc6c54770bb6cbf4923ddb3073775664a538eb" exitCode=0 Nov 02 01:48:59.776629 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:59.776498 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp" event={"ID":"8f9b0669-291d-43fa-9511-d74948b03684","Type":"ContainerDied","Data":"f3bbaed735d3a00f7bdd6497f8dc6c54770bb6cbf4923ddb3073775664a538eb"} Nov 02 01:48:59.776629 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:59.776552 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp" event={"ID":"8f9b0669-291d-43fa-9511-d74948b03684","Type":"ContainerDied","Data":"f8f08b2d8126733ec5e6122b23ffd39f98f510175c65804eb2d5de0f95ae0fab"} Nov 02 01:48:59.776629 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:59.776583 4427 scope.go:122] "RemoveContainer" containerID="f3bbaed735d3a00f7bdd6497f8dc6c54770bb6cbf4923ddb3073775664a538eb" Nov 02 01:48:59.776911 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:59.776756 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp" Nov 02 01:48:59.792925 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:59.792894 4427 scope.go:122] "RemoveContainer" containerID="1a94ae972be836c0ad68ee977315577de6860497757ed3ef245783a084447f50" Nov 02 01:48:59.798312 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:59.798284 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp"] Nov 02 01:48:59.802819 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:59.802794 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-0e06f2-openshift-default-5f45987d85-djzvp"] Nov 02 01:48:59.804429 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:59.804408 4427 scope.go:122] "RemoveContainer" containerID="f3bbaed735d3a00f7bdd6497f8dc6c54770bb6cbf4923ddb3073775664a538eb" Nov 02 01:48:59.804774 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:48:59.804755 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3bbaed735d3a00f7bdd6497f8dc6c54770bb6cbf4923ddb3073775664a538eb\": container with ID starting with f3bbaed735d3a00f7bdd6497f8dc6c54770bb6cbf4923ddb3073775664a538eb not found: ID does not exist" containerID="f3bbaed735d3a00f7bdd6497f8dc6c54770bb6cbf4923ddb3073775664a538eb" Nov 02 01:48:59.804824 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:59.804784 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3bbaed735d3a00f7bdd6497f8dc6c54770bb6cbf4923ddb3073775664a538eb"} err="failed to get container status \"f3bbaed735d3a00f7bdd6497f8dc6c54770bb6cbf4923ddb3073775664a538eb\": rpc error: code = NotFound desc = could not find container \"f3bbaed735d3a00f7bdd6497f8dc6c54770bb6cbf4923ddb3073775664a538eb\": container with ID starting with f3bbaed735d3a00f7bdd6497f8dc6c54770bb6cbf4923ddb3073775664a538eb not found: ID does not exist" Nov 02 01:48:59.804824 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:59.804802 4427 scope.go:122] "RemoveContainer" containerID="1a94ae972be836c0ad68ee977315577de6860497757ed3ef245783a084447f50" Nov 02 01:48:59.805099 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:48:59.805078 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a94ae972be836c0ad68ee977315577de6860497757ed3ef245783a084447f50\": container with ID starting with 1a94ae972be836c0ad68ee977315577de6860497757ed3ef245783a084447f50 not found: ID does not exist" containerID="1a94ae972be836c0ad68ee977315577de6860497757ed3ef245783a084447f50" Nov 02 01:48:59.805137 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:48:59.805105 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a94ae972be836c0ad68ee977315577de6860497757ed3ef245783a084447f50"} err="failed to get container status \"1a94ae972be836c0ad68ee977315577de6860497757ed3ef245783a084447f50\": rpc error: code = NotFound desc = could not find container \"1a94ae972be836c0ad68ee977315577de6860497757ed3ef245783a084447f50\": container with ID starting with 1a94ae972be836c0ad68ee977315577de6860497757ed3ef245783a084447f50 not found: ID does not exist" Nov 02 01:49:00.309650 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:00.309616 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d4a0-6595f44dff-m47jt"] Nov 02 01:49:01.533908 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:01.533874 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f9b0669-291d-43fa-9511-d74948b03684" path="/var/lib/kubelet/pods/8f9b0669-291d-43fa-9511-d74948b03684/volumes" Nov 02 01:49:01.585482 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:49:01.585444 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:49:03.100877 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:03.100829 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e06f2-855bdc6447-dxbs5" podUID="aceec71b-ffcb-47e7-964e-84bc8405f2de" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 01:49:03.100877 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:03.100867 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e06f2-64fc8f586c-8sw9n" podUID="79fb94dd-4426-43ed-af58-cfa762e77b13" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 01:49:04.901101 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:04.900471 4427 generic.go:356] "Generic (PLEG): container finished" podID="f31dc5a1-ea54-4858-ac4c-2b990192ee4c" containerID="b9a3277d2256fe4b6a16984a340fd19b7a21f7eb6724b4912989a4d1bba47422" exitCode=0 Nov 02 01:49:04.901101 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:04.900700 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22" event={"ID":"f31dc5a1-ea54-4858-ac4c-2b990192ee4c","Type":"ContainerDied","Data":"b9a3277d2256fe4b6a16984a340fd19b7a21f7eb6724b4912989a4d1bba47422"} Nov 02 01:49:04.901101 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:04.900828 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22" event={"ID":"f31dc5a1-ea54-4858-ac4c-2b990192ee4c","Type":"ContainerDied","Data":"40bc9baa4287766a5917407c753f37769ed9b41e8f2f938d5d1b7bb0eb1ba25d"} Nov 02 01:49:04.901101 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:04.900856 4427 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40bc9baa4287766a5917407c753f37769ed9b41e8f2f938d5d1b7bb0eb1ba25d" Nov 02 01:49:04.912410 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:04.908945 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22" Nov 02 01:49:04.983874 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:04.983809 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f31dc5a1-ea54-4858-ac4c-2b990192ee4c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f31dc5a1-ea54-4858-ac4c-2b990192ee4c-istio-envoy\") pod \"f31dc5a1-ea54-4858-ac4c-2b990192ee4c\" (UID: \"f31dc5a1-ea54-4858-ac4c-2b990192ee4c\") " Nov 02 01:49:04.984075 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:04.983961 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f31dc5a1-ea54-4858-ac4c-2b990192ee4c-kube-api-access-ns6dn\" (UniqueName: \"kubernetes.io/projected/f31dc5a1-ea54-4858-ac4c-2b990192ee4c-kube-api-access-ns6dn\") pod \"f31dc5a1-ea54-4858-ac4c-2b990192ee4c\" (UID: \"f31dc5a1-ea54-4858-ac4c-2b990192ee4c\") " Nov 02 01:49:04.984075 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:04.984000 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f31dc5a1-ea54-4858-ac4c-2b990192ee4c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f31dc5a1-ea54-4858-ac4c-2b990192ee4c-credential-socket\") pod \"f31dc5a1-ea54-4858-ac4c-2b990192ee4c\" (UID: \"f31dc5a1-ea54-4858-ac4c-2b990192ee4c\") " Nov 02 01:49:04.984075 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:04.984029 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f31dc5a1-ea54-4858-ac4c-2b990192ee4c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f31dc5a1-ea54-4858-ac4c-2b990192ee4c-istio-data\") pod \"f31dc5a1-ea54-4858-ac4c-2b990192ee4c\" (UID: \"f31dc5a1-ea54-4858-ac4c-2b990192ee4c\") " Nov 02 01:49:04.984222 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:04.984124 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f31dc5a1-ea54-4858-ac4c-2b990192ee4c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f31dc5a1-ea54-4858-ac4c-2b990192ee4c-workload-socket\") pod \"f31dc5a1-ea54-4858-ac4c-2b990192ee4c\" (UID: \"f31dc5a1-ea54-4858-ac4c-2b990192ee4c\") " Nov 02 01:49:04.984222 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:04.984157 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f31dc5a1-ea54-4858-ac4c-2b990192ee4c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f31dc5a1-ea54-4858-ac4c-2b990192ee4c-workload-certs\") pod \"f31dc5a1-ea54-4858-ac4c-2b990192ee4c\" (UID: \"f31dc5a1-ea54-4858-ac4c-2b990192ee4c\") " Nov 02 01:49:04.984222 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:04.984189 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f31dc5a1-ea54-4858-ac4c-2b990192ee4c-istio-token\" (UniqueName: \"kubernetes.io/projected/f31dc5a1-ea54-4858-ac4c-2b990192ee4c-istio-token\") pod \"f31dc5a1-ea54-4858-ac4c-2b990192ee4c\" (UID: \"f31dc5a1-ea54-4858-ac4c-2b990192ee4c\") " Nov 02 01:49:04.984429 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:04.984264 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f31dc5a1-ea54-4858-ac4c-2b990192ee4c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f31dc5a1-ea54-4858-ac4c-2b990192ee4c-istiod-ca-cert\") pod \"f31dc5a1-ea54-4858-ac4c-2b990192ee4c\" (UID: \"f31dc5a1-ea54-4858-ac4c-2b990192ee4c\") " Nov 02 01:49:04.984429 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:04.984312 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f31dc5a1-ea54-4858-ac4c-2b990192ee4c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f31dc5a1-ea54-4858-ac4c-2b990192ee4c-istio-podinfo\") pod \"f31dc5a1-ea54-4858-ac4c-2b990192ee4c\" (UID: \"f31dc5a1-ea54-4858-ac4c-2b990192ee4c\") " Nov 02 01:49:04.984429 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:04.984365 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f31dc5a1-ea54-4858-ac4c-2b990192ee4c-istio-data" pod "f31dc5a1-ea54-4858-ac4c-2b990192ee4c" (UID: "f31dc5a1-ea54-4858-ac4c-2b990192ee4c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:04.984606 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:04.984494 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f31dc5a1-ea54-4858-ac4c-2b990192ee4c-workload-socket" pod "f31dc5a1-ea54-4858-ac4c-2b990192ee4c" (UID: "f31dc5a1-ea54-4858-ac4c-2b990192ee4c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:04.984926 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:04.984900 4427 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f31dc5a1-ea54-4858-ac4c-2b990192ee4c-workload-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:04.985012 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:04.984933 4427 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f31dc5a1-ea54-4858-ac4c-2b990192ee4c-istio-data\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:04.985012 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:04.984941 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f31dc5a1-ea54-4858-ac4c-2b990192ee4c-credential-socket" pod "f31dc5a1-ea54-4858-ac4c-2b990192ee4c" (UID: "f31dc5a1-ea54-4858-ac4c-2b990192ee4c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:04.985148 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:04.985127 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f31dc5a1-ea54-4858-ac4c-2b990192ee4c-istiod-ca-cert" pod "f31dc5a1-ea54-4858-ac4c-2b990192ee4c" (UID: "f31dc5a1-ea54-4858-ac4c-2b990192ee4c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:49:04.985358 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:04.985339 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f31dc5a1-ea54-4858-ac4c-2b990192ee4c-workload-certs" pod "f31dc5a1-ea54-4858-ac4c-2b990192ee4c" (UID: "f31dc5a1-ea54-4858-ac4c-2b990192ee4c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:04.987062 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:04.987042 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f31dc5a1-ea54-4858-ac4c-2b990192ee4c-kube-api-access-ns6dn" pod "f31dc5a1-ea54-4858-ac4c-2b990192ee4c" (UID: "f31dc5a1-ea54-4858-ac4c-2b990192ee4c"). InnerVolumeSpecName "kube-api-access-ns6dn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:49:04.987180 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:04.987062 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f31dc5a1-ea54-4858-ac4c-2b990192ee4c-istio-envoy" pod "f31dc5a1-ea54-4858-ac4c-2b990192ee4c" (UID: "f31dc5a1-ea54-4858-ac4c-2b990192ee4c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:04.987180 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:04.987062 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f31dc5a1-ea54-4858-ac4c-2b990192ee4c-istio-token" pod "f31dc5a1-ea54-4858-ac4c-2b990192ee4c" (UID: "f31dc5a1-ea54-4858-ac4c-2b990192ee4c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:49:04.987409 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:04.987365 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f31dc5a1-ea54-4858-ac4c-2b990192ee4c-istio-podinfo" pod "f31dc5a1-ea54-4858-ac4c-2b990192ee4c" (UID: "f31dc5a1-ea54-4858-ac4c-2b990192ee4c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:49:05.086060 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:05.085971 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ns6dn\" (UniqueName: \"kubernetes.io/projected/f31dc5a1-ea54-4858-ac4c-2b990192ee4c-kube-api-access-ns6dn\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:05.086060 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:05.086003 4427 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f31dc5a1-ea54-4858-ac4c-2b990192ee4c-credential-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:05.086060 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:05.086016 4427 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f31dc5a1-ea54-4858-ac4c-2b990192ee4c-workload-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:05.086060 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:05.086029 4427 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f31dc5a1-ea54-4858-ac4c-2b990192ee4c-istio-token\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:05.086060 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:05.086043 4427 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f31dc5a1-ea54-4858-ac4c-2b990192ee4c-istiod-ca-cert\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:05.086060 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:05.086051 4427 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f31dc5a1-ea54-4858-ac4c-2b990192ee4c-istio-podinfo\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:05.086060 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:05.086060 4427 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f31dc5a1-ea54-4858-ac4c-2b990192ee4c-istio-envoy\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:05.310012 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:05.309944 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d4a0-6595f44dff-m47jt" podUID="fca29702-c34c-4933-a93a-143c5481ba3a" containerName="payload-pre-processing" containerID="cri-o://2e3970879f8b3a231c8a1dbb7cc862a378562a3558e2b134903caf126aac381b" gracePeriod=25 Nov 02 01:49:05.483626 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:05.483587 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d4a0-6595f44dff-m47jt" podUID="fca29702-c34c-4933-a93a-143c5481ba3a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 01:49:05.907129 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:05.907093 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22" Nov 02 01:49:05.927659 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:05.927608 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22"] Nov 02 01:49:05.931613 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:05.931582 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-eaec8c-openshift-default-5f4f959567-czp22"] Nov 02 01:49:06.915457 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:06.915423 4427 generic.go:356] "Generic (PLEG): container finished" podID="79fb94dd-4426-43ed-af58-cfa762e77b13" containerID="d7c4ee8f3187db030a01706818be0cad0f93975643eef0c147bd0ba2ba6e98dc" exitCode=0 Nov 02 01:49:06.915945 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:06.915493 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e06f2-64fc8f586c-8sw9n" event={"ID":"79fb94dd-4426-43ed-af58-cfa762e77b13","Type":"ContainerDied","Data":"d7c4ee8f3187db030a01706818be0cad0f93975643eef0c147bd0ba2ba6e98dc"} Nov 02 01:49:06.924600 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:06.924555 4427 generic.go:356] "Generic (PLEG): container finished" podID="aceec71b-ffcb-47e7-964e-84bc8405f2de" containerID="9f85df429f76202a4141848bceefc109dd8f3ee26d5e6b2333455bf8f40c719f" exitCode=0 Nov 02 01:49:06.924798 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:06.924674 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e06f2-855bdc6447-dxbs5" event={"ID":"aceec71b-ffcb-47e7-964e-84bc8405f2de","Type":"ContainerDied","Data":"9f85df429f76202a4141848bceefc109dd8f3ee26d5e6b2333455bf8f40c719f"} Nov 02 01:49:07.099295 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:07.099263 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e06f2-64fc8f586c-8sw9n" Nov 02 01:49:07.127621 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:07.127584 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e06f2-855bdc6447-dxbs5" Nov 02 01:49:07.216484 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:07.216447 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aceec71b-ffcb-47e7-964e-84bc8405f2de-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aceec71b-ffcb-47e7-964e-84bc8405f2de-plugins-config-volume\") pod \"aceec71b-ffcb-47e7-964e-84bc8405f2de\" (UID: \"aceec71b-ffcb-47e7-964e-84bc8405f2de\") " Nov 02 01:49:07.216691 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:07.216509 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aceec71b-ffcb-47e7-964e-84bc8405f2de-kube-api-access-2h2q6\" (UniqueName: \"kubernetes.io/projected/aceec71b-ffcb-47e7-964e-84bc8405f2de-kube-api-access-2h2q6\") pod \"aceec71b-ffcb-47e7-964e-84bc8405f2de\" (UID: \"aceec71b-ffcb-47e7-964e-84bc8405f2de\") " Nov 02 01:49:07.216691 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:07.216528 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79fb94dd-4426-43ed-af58-cfa762e77b13-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79fb94dd-4426-43ed-af58-cfa762e77b13-plugins-config-volume\") pod \"79fb94dd-4426-43ed-af58-cfa762e77b13\" (UID: \"79fb94dd-4426-43ed-af58-cfa762e77b13\") " Nov 02 01:49:07.216691 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:07.216658 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79fb94dd-4426-43ed-af58-cfa762e77b13-kube-api-access-mbtq7\" (UniqueName: \"kubernetes.io/projected/79fb94dd-4426-43ed-af58-cfa762e77b13-kube-api-access-mbtq7\") pod \"79fb94dd-4426-43ed-af58-cfa762e77b13\" (UID: \"79fb94dd-4426-43ed-af58-cfa762e77b13\") " Nov 02 01:49:07.216987 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:07.216968 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79fb94dd-4426-43ed-af58-cfa762e77b13-plugins-config-volume" pod "79fb94dd-4426-43ed-af58-cfa762e77b13" (UID: "79fb94dd-4426-43ed-af58-cfa762e77b13"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:49:07.217061 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:07.216965 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aceec71b-ffcb-47e7-964e-84bc8405f2de-plugins-config-volume" pod "aceec71b-ffcb-47e7-964e-84bc8405f2de" (UID: "aceec71b-ffcb-47e7-964e-84bc8405f2de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:49:07.217257 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:07.217236 4427 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aceec71b-ffcb-47e7-964e-84bc8405f2de-plugins-config-volume\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:07.217325 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:07.217262 4427 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79fb94dd-4426-43ed-af58-cfa762e77b13-plugins-config-volume\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:07.219012 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:07.218988 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aceec71b-ffcb-47e7-964e-84bc8405f2de-kube-api-access-2h2q6" pod "aceec71b-ffcb-47e7-964e-84bc8405f2de" (UID: "aceec71b-ffcb-47e7-964e-84bc8405f2de"). InnerVolumeSpecName "kube-api-access-2h2q6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:49:07.219468 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:07.219449 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79fb94dd-4426-43ed-af58-cfa762e77b13-kube-api-access-mbtq7" pod "79fb94dd-4426-43ed-af58-cfa762e77b13" (UID: "79fb94dd-4426-43ed-af58-cfa762e77b13"). InnerVolumeSpecName "kube-api-access-mbtq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:49:07.318277 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:07.318191 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbtq7\" (UniqueName: \"kubernetes.io/projected/79fb94dd-4426-43ed-af58-cfa762e77b13-kube-api-access-mbtq7\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:07.318277 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:07.318219 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h2q6\" (UniqueName: \"kubernetes.io/projected/aceec71b-ffcb-47e7-964e-84bc8405f2de-kube-api-access-2h2q6\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:07.533347 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:07.533311 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f31dc5a1-ea54-4858-ac4c-2b990192ee4c" path="/var/lib/kubelet/pods/f31dc5a1-ea54-4858-ac4c-2b990192ee4c/volumes" Nov 02 01:49:07.932978 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:07.932946 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e06f2-64fc8f586c-8sw9n" Nov 02 01:49:07.933488 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:07.932946 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e06f2-64fc8f586c-8sw9n" event={"ID":"79fb94dd-4426-43ed-af58-cfa762e77b13","Type":"ContainerDied","Data":"2c0c79d5c8b8a7f0778c16d8955211a18e42875f0fdc8c6a894dca9edd8aa9db"} Nov 02 01:49:07.933488 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:07.933099 4427 scope.go:122] "RemoveContainer" containerID="d7c4ee8f3187db030a01706818be0cad0f93975643eef0c147bd0ba2ba6e98dc" Nov 02 01:49:07.934595 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:07.934572 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e06f2-855bdc6447-dxbs5" event={"ID":"aceec71b-ffcb-47e7-964e-84bc8405f2de","Type":"ContainerDied","Data":"f48343db89be37693f4127e0d60e698b96dbca46122a2f2c91efceb66577ac3d"} Nov 02 01:49:07.934686 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:07.934585 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e06f2-855bdc6447-dxbs5" Nov 02 01:49:07.943910 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:07.943891 4427 scope.go:122] "RemoveContainer" containerID="9f85df429f76202a4141848bceefc109dd8f3ee26d5e6b2333455bf8f40c719f" Nov 02 01:49:07.957418 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:07.957393 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-0e06f2-64fc8f586c-8sw9n"] Nov 02 01:49:07.962296 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:07.962268 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-0e06f2-64fc8f586c-8sw9n"] Nov 02 01:49:07.974787 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:07.974759 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e06f2-855bdc6447-dxbs5"] Nov 02 01:49:07.981393 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:07.981350 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e06f2-855bdc6447-dxbs5"] Nov 02 01:49:09.533681 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:09.533647 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79fb94dd-4426-43ed-af58-cfa762e77b13" path="/var/lib/kubelet/pods/79fb94dd-4426-43ed-af58-cfa762e77b13/volumes" Nov 02 01:49:09.534072 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:09.534058 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aceec71b-ffcb-47e7-964e-84bc8405f2de" path="/var/lib/kubelet/pods/aceec71b-ffcb-47e7-964e-84bc8405f2de/volumes" Nov 02 01:49:10.142001 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:10.141949 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0f32d6-647fd89d8c-v79jr"] Nov 02 01:49:12.380358 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:12.380323 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp"] Nov 02 01:49:12.381336 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:12.381298 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp" podUID="1d6df883-72f0-4727-acbc-72bb9bd02f05" containerName="istio-proxy" containerID="cri-o://b850b4e46248ca4510211fae40e97cf3e098413ce855ed741fc0742a4d94a80d" gracePeriod=30 Nov 02 01:49:15.142448 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:15.142364 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0f32d6-647fd89d8c-v79jr" podUID="17e05811-e6f0-4c5b-be43-d39429d1af11" containerName="payload-pre-processing" containerID="cri-o://4f20ca6d1ed8b5e95fc91ddac56434196c7a2daae0e290eb30bf9d6e6a38f59c" gracePeriod=25 Nov 02 01:49:15.483019 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:15.482927 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d4a0-6595f44dff-m47jt" podUID="fca29702-c34c-4933-a93a-143c5481ba3a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 01:49:17.302926 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:17.302892 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0f32d6-647fd89d8c-v79jr" podUID="17e05811-e6f0-4c5b-be43-d39429d1af11" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 01:49:17.669322 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:17.669273 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp" Nov 02 01:49:17.762308 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:17.762271 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1d6df883-72f0-4727-acbc-72bb9bd02f05-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d6df883-72f0-4727-acbc-72bb9bd02f05-istio-podinfo\") pod \"1d6df883-72f0-4727-acbc-72bb9bd02f05\" (UID: \"1d6df883-72f0-4727-acbc-72bb9bd02f05\") " Nov 02 01:49:17.762523 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:17.762358 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d6df883-72f0-4727-acbc-72bb9bd02f05-istio-token\" (UniqueName: \"kubernetes.io/projected/1d6df883-72f0-4727-acbc-72bb9bd02f05-istio-token\") pod \"1d6df883-72f0-4727-acbc-72bb9bd02f05\" (UID: \"1d6df883-72f0-4727-acbc-72bb9bd02f05\") " Nov 02 01:49:17.762523 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:17.762454 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d6df883-72f0-4727-acbc-72bb9bd02f05-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d6df883-72f0-4727-acbc-72bb9bd02f05-workload-certs\") pod \"1d6df883-72f0-4727-acbc-72bb9bd02f05\" (UID: \"1d6df883-72f0-4727-acbc-72bb9bd02f05\") " Nov 02 01:49:17.762670 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:17.762528 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d6df883-72f0-4727-acbc-72bb9bd02f05-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d6df883-72f0-4727-acbc-72bb9bd02f05-istiod-ca-cert\") pod \"1d6df883-72f0-4727-acbc-72bb9bd02f05\" (UID: \"1d6df883-72f0-4727-acbc-72bb9bd02f05\") " Nov 02 01:49:17.762670 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:17.762573 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d6df883-72f0-4727-acbc-72bb9bd02f05-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d6df883-72f0-4727-acbc-72bb9bd02f05-workload-socket\") pod \"1d6df883-72f0-4727-acbc-72bb9bd02f05\" (UID: \"1d6df883-72f0-4727-acbc-72bb9bd02f05\") " Nov 02 01:49:17.762670 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:17.762625 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d6df883-72f0-4727-acbc-72bb9bd02f05-kube-api-access-xcj5t\" (UniqueName: \"kubernetes.io/projected/1d6df883-72f0-4727-acbc-72bb9bd02f05-kube-api-access-xcj5t\") pod \"1d6df883-72f0-4727-acbc-72bb9bd02f05\" (UID: \"1d6df883-72f0-4727-acbc-72bb9bd02f05\") " Nov 02 01:49:17.762670 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:17.762652 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d6df883-72f0-4727-acbc-72bb9bd02f05-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d6df883-72f0-4727-acbc-72bb9bd02f05-istio-envoy\") pod \"1d6df883-72f0-4727-acbc-72bb9bd02f05\" (UID: \"1d6df883-72f0-4727-acbc-72bb9bd02f05\") " Nov 02 01:49:17.762893 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:17.762750 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d6df883-72f0-4727-acbc-72bb9bd02f05-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d6df883-72f0-4727-acbc-72bb9bd02f05-istio-data\") pod \"1d6df883-72f0-4727-acbc-72bb9bd02f05\" (UID: \"1d6df883-72f0-4727-acbc-72bb9bd02f05\") " Nov 02 01:49:17.762893 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:17.762826 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d6df883-72f0-4727-acbc-72bb9bd02f05-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d6df883-72f0-4727-acbc-72bb9bd02f05-credential-socket\") pod \"1d6df883-72f0-4727-acbc-72bb9bd02f05\" (UID: \"1d6df883-72f0-4727-acbc-72bb9bd02f05\") " Nov 02 01:49:17.763401 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:17.763148 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d6df883-72f0-4727-acbc-72bb9bd02f05-istiod-ca-cert" pod "1d6df883-72f0-4727-acbc-72bb9bd02f05" (UID: "1d6df883-72f0-4727-acbc-72bb9bd02f05"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:49:17.763744 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:17.763575 4427 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d6df883-72f0-4727-acbc-72bb9bd02f05-istiod-ca-cert\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:17.763744 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:17.763578 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d6df883-72f0-4727-acbc-72bb9bd02f05-workload-socket" pod "1d6df883-72f0-4727-acbc-72bb9bd02f05" (UID: "1d6df883-72f0-4727-acbc-72bb9bd02f05"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:17.763744 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:17.763626 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d6df883-72f0-4727-acbc-72bb9bd02f05-credential-socket" pod "1d6df883-72f0-4727-acbc-72bb9bd02f05" (UID: "1d6df883-72f0-4727-acbc-72bb9bd02f05"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:17.763962 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:17.763815 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d6df883-72f0-4727-acbc-72bb9bd02f05-istio-data" pod "1d6df883-72f0-4727-acbc-72bb9bd02f05" (UID: "1d6df883-72f0-4727-acbc-72bb9bd02f05"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:17.764027 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:17.763996 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d6df883-72f0-4727-acbc-72bb9bd02f05-workload-certs" pod "1d6df883-72f0-4727-acbc-72bb9bd02f05" (UID: "1d6df883-72f0-4727-acbc-72bb9bd02f05"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:17.765413 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:17.765335 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d6df883-72f0-4727-acbc-72bb9bd02f05-istio-podinfo" pod "1d6df883-72f0-4727-acbc-72bb9bd02f05" (UID: "1d6df883-72f0-4727-acbc-72bb9bd02f05"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:49:17.765845 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:17.765815 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d6df883-72f0-4727-acbc-72bb9bd02f05-istio-token" pod "1d6df883-72f0-4727-acbc-72bb9bd02f05" (UID: "1d6df883-72f0-4727-acbc-72bb9bd02f05"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:49:17.766806 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:17.766773 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d6df883-72f0-4727-acbc-72bb9bd02f05-istio-envoy" pod "1d6df883-72f0-4727-acbc-72bb9bd02f05" (UID: "1d6df883-72f0-4727-acbc-72bb9bd02f05"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:17.767286 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:17.767263 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d6df883-72f0-4727-acbc-72bb9bd02f05-kube-api-access-xcj5t" pod "1d6df883-72f0-4727-acbc-72bb9bd02f05" (UID: "1d6df883-72f0-4727-acbc-72bb9bd02f05"). InnerVolumeSpecName "kube-api-access-xcj5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:49:17.865218 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:17.865188 4427 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d6df883-72f0-4727-acbc-72bb9bd02f05-istio-data\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:17.865218 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:17.865217 4427 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d6df883-72f0-4727-acbc-72bb9bd02f05-credential-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:17.865453 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:17.865226 4427 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d6df883-72f0-4727-acbc-72bb9bd02f05-istio-podinfo\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:17.865453 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:17.865236 4427 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d6df883-72f0-4727-acbc-72bb9bd02f05-istio-token\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:17.865453 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:17.865247 4427 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d6df883-72f0-4727-acbc-72bb9bd02f05-workload-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:17.865453 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:17.865255 4427 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d6df883-72f0-4727-acbc-72bb9bd02f05-workload-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:17.865453 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:17.865266 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcj5t\" (UniqueName: \"kubernetes.io/projected/1d6df883-72f0-4727-acbc-72bb9bd02f05-kube-api-access-xcj5t\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:17.865453 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:17.865275 4427 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d6df883-72f0-4727-acbc-72bb9bd02f05-istio-envoy\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:17.954853 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:17.954815 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr"] Nov 02 01:49:18.007295 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:18.007263 4427 generic.go:356] "Generic (PLEG): container finished" podID="1d6df883-72f0-4727-acbc-72bb9bd02f05" containerID="b850b4e46248ca4510211fae40e97cf3e098413ce855ed741fc0742a4d94a80d" exitCode=0 Nov 02 01:49:18.007483 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:18.007318 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp" Nov 02 01:49:18.007483 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:18.007329 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp" event={"ID":"1d6df883-72f0-4727-acbc-72bb9bd02f05","Type":"ContainerDied","Data":"b850b4e46248ca4510211fae40e97cf3e098413ce855ed741fc0742a4d94a80d"} Nov 02 01:49:18.007483 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:18.007387 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp" event={"ID":"1d6df883-72f0-4727-acbc-72bb9bd02f05","Type":"ContainerDied","Data":"e8a458a7a498a9070ba2c9f8400cddcbbc1b9dc616bf68b3eff64f2f3337c7cf"} Nov 02 01:49:18.007483 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:18.007415 4427 scope.go:122] "RemoveContainer" containerID="b850b4e46248ca4510211fae40e97cf3e098413ce855ed741fc0742a4d94a80d" Nov 02 01:49:18.021454 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:18.021431 4427 scope.go:122] "RemoveContainer" containerID="1998796f0577d5eb3cec5d2bcc5e6bcc70971c4401f42cf0951a7e59b1e3cc7c" Nov 02 01:49:18.032079 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:18.032050 4427 scope.go:122] "RemoveContainer" containerID="b850b4e46248ca4510211fae40e97cf3e098413ce855ed741fc0742a4d94a80d" Nov 02 01:49:18.032421 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:49:18.032364 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b850b4e46248ca4510211fae40e97cf3e098413ce855ed741fc0742a4d94a80d\": container with ID starting with b850b4e46248ca4510211fae40e97cf3e098413ce855ed741fc0742a4d94a80d not found: ID does not exist" containerID="b850b4e46248ca4510211fae40e97cf3e098413ce855ed741fc0742a4d94a80d" Nov 02 01:49:18.032532 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:18.032435 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b850b4e46248ca4510211fae40e97cf3e098413ce855ed741fc0742a4d94a80d"} err="failed to get container status \"b850b4e46248ca4510211fae40e97cf3e098413ce855ed741fc0742a4d94a80d\": rpc error: code = NotFound desc = could not find container \"b850b4e46248ca4510211fae40e97cf3e098413ce855ed741fc0742a4d94a80d\": container with ID starting with b850b4e46248ca4510211fae40e97cf3e098413ce855ed741fc0742a4d94a80d not found: ID does not exist" Nov 02 01:49:18.032532 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:18.032460 4427 scope.go:122] "RemoveContainer" containerID="1998796f0577d5eb3cec5d2bcc5e6bcc70971c4401f42cf0951a7e59b1e3cc7c" Nov 02 01:49:18.032774 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:49:18.032751 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1998796f0577d5eb3cec5d2bcc5e6bcc70971c4401f42cf0951a7e59b1e3cc7c\": container with ID starting with 1998796f0577d5eb3cec5d2bcc5e6bcc70971c4401f42cf0951a7e59b1e3cc7c not found: ID does not exist" containerID="1998796f0577d5eb3cec5d2bcc5e6bcc70971c4401f42cf0951a7e59b1e3cc7c" Nov 02 01:49:18.032814 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:18.032785 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1998796f0577d5eb3cec5d2bcc5e6bcc70971c4401f42cf0951a7e59b1e3cc7c"} err="failed to get container status \"1998796f0577d5eb3cec5d2bcc5e6bcc70971c4401f42cf0951a7e59b1e3cc7c\": rpc error: code = NotFound desc = could not find container \"1998796f0577d5eb3cec5d2bcc5e6bcc70971c4401f42cf0951a7e59b1e3cc7c\": container with ID starting with 1998796f0577d5eb3cec5d2bcc5e6bcc70971c4401f42cf0951a7e59b1e3cc7c not found: ID does not exist" Nov 02 01:49:18.033028 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:18.033010 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp"] Nov 02 01:49:18.039332 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:18.039306 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c8d4a0-openshift-default-888d8bfc-gr8hp"] Nov 02 01:49:19.533848 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:19.533808 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d6df883-72f0-4727-acbc-72bb9bd02f05" path="/var/lib/kubelet/pods/1d6df883-72f0-4727-acbc-72bb9bd02f05/volumes" Nov 02 01:49:20.863404 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:20.856068 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6"] Nov 02 01:49:20.863404 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:20.856315 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6" podUID="392681ce-fe14-4518-9697-efb94f82efdc" containerName="istio-proxy" containerID="cri-o://44ced00972441350c17d5df842a755ef8ecae0ab1a661e0b090a4e0a376f9591" gracePeriod=30 Nov 02 01:49:25.482433 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:49:25.482401 4427 logging.go:55] [core] [Channel #3544 SubChannel #3545]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.149:50052", ServerName: "10.133.0.149:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.149:50052: connect: connection refused" Nov 02 01:49:25.593532 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:25.593505 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d4a0-6595f44dff-m47jt" Nov 02 01:49:25.653867 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:25.653838 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fca29702-c34c-4933-a93a-143c5481ba3a-kube-api-access-sp6dn\" (UniqueName: \"kubernetes.io/projected/fca29702-c34c-4933-a93a-143c5481ba3a-kube-api-access-sp6dn\") pod \"fca29702-c34c-4933-a93a-143c5481ba3a\" (UID: \"fca29702-c34c-4933-a93a-143c5481ba3a\") " Nov 02 01:49:25.654120 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:25.654004 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fca29702-c34c-4933-a93a-143c5481ba3a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fca29702-c34c-4933-a93a-143c5481ba3a-plugins-config-volume\") pod \"fca29702-c34c-4933-a93a-143c5481ba3a\" (UID: \"fca29702-c34c-4933-a93a-143c5481ba3a\") " Nov 02 01:49:25.654442 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:25.654410 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fca29702-c34c-4933-a93a-143c5481ba3a-plugins-config-volume" pod "fca29702-c34c-4933-a93a-143c5481ba3a" (UID: "fca29702-c34c-4933-a93a-143c5481ba3a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:49:25.656291 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:25.656265 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fca29702-c34c-4933-a93a-143c5481ba3a-kube-api-access-sp6dn" pod "fca29702-c34c-4933-a93a-143c5481ba3a" (UID: "fca29702-c34c-4933-a93a-143c5481ba3a"). InnerVolumeSpecName "kube-api-access-sp6dn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:49:25.755400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:25.755345 4427 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fca29702-c34c-4933-a93a-143c5481ba3a-plugins-config-volume\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:25.755400 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:25.755392 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sp6dn\" (UniqueName: \"kubernetes.io/projected/fca29702-c34c-4933-a93a-143c5481ba3a-kube-api-access-sp6dn\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:26.066410 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:26.066362 4427 generic.go:356] "Generic (PLEG): container finished" podID="392681ce-fe14-4518-9697-efb94f82efdc" containerID="44ced00972441350c17d5df842a755ef8ecae0ab1a661e0b090a4e0a376f9591" exitCode=0 Nov 02 01:49:26.066650 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:26.066413 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6" event={"ID":"392681ce-fe14-4518-9697-efb94f82efdc","Type":"ContainerDied","Data":"44ced00972441350c17d5df842a755ef8ecae0ab1a661e0b090a4e0a376f9591"} Nov 02 01:49:26.068604 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:26.068571 4427 generic.go:356] "Generic (PLEG): container finished" podID="fca29702-c34c-4933-a93a-143c5481ba3a" containerID="2e3970879f8b3a231c8a1dbb7cc862a378562a3558e2b134903caf126aac381b" exitCode=0 Nov 02 01:49:26.068742 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:26.068627 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d4a0-6595f44dff-m47jt" Nov 02 01:49:26.068742 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:26.068651 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d4a0-6595f44dff-m47jt" event={"ID":"fca29702-c34c-4933-a93a-143c5481ba3a","Type":"ContainerDied","Data":"2e3970879f8b3a231c8a1dbb7cc862a378562a3558e2b134903caf126aac381b"} Nov 02 01:49:26.068742 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:26.068705 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d4a0-6595f44dff-m47jt" event={"ID":"fca29702-c34c-4933-a93a-143c5481ba3a","Type":"ContainerDied","Data":"3fa4ef2c2f34bad9fdbb972fff7de519e18ee24dae029e0e980c5538b84311bd"} Nov 02 01:49:26.068742 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:26.068726 4427 scope.go:122] "RemoveContainer" containerID="2e3970879f8b3a231c8a1dbb7cc862a378562a3558e2b134903caf126aac381b" Nov 02 01:49:26.095538 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:26.095513 4427 scope.go:122] "RemoveContainer" containerID="2e3970879f8b3a231c8a1dbb7cc862a378562a3558e2b134903caf126aac381b" Nov 02 01:49:26.095932 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:49:26.095908 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e3970879f8b3a231c8a1dbb7cc862a378562a3558e2b134903caf126aac381b\": container with ID starting with 2e3970879f8b3a231c8a1dbb7cc862a378562a3558e2b134903caf126aac381b not found: ID does not exist" containerID="2e3970879f8b3a231c8a1dbb7cc862a378562a3558e2b134903caf126aac381b" Nov 02 01:49:26.096026 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:26.095943 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e3970879f8b3a231c8a1dbb7cc862a378562a3558e2b134903caf126aac381b"} err="failed to get container status \"2e3970879f8b3a231c8a1dbb7cc862a378562a3558e2b134903caf126aac381b\": rpc error: code = NotFound desc = could not find container \"2e3970879f8b3a231c8a1dbb7cc862a378562a3558e2b134903caf126aac381b\": container with ID starting with 2e3970879f8b3a231c8a1dbb7cc862a378562a3558e2b134903caf126aac381b not found: ID does not exist" Nov 02 01:49:26.114509 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:26.114473 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d4a0-6595f44dff-m47jt"] Nov 02 01:49:26.120351 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:26.120321 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d4a0-6595f44dff-m47jt"] Nov 02 01:49:26.150521 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:26.150495 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6" Nov 02 01:49:26.159456 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:26.159428 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/392681ce-fe14-4518-9697-efb94f82efdc-istio-token\" (UniqueName: \"kubernetes.io/projected/392681ce-fe14-4518-9697-efb94f82efdc-istio-token\") pod \"392681ce-fe14-4518-9697-efb94f82efdc\" (UID: \"392681ce-fe14-4518-9697-efb94f82efdc\") " Nov 02 01:49:26.159617 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:26.159544 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/392681ce-fe14-4518-9697-efb94f82efdc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/392681ce-fe14-4518-9697-efb94f82efdc-workload-socket\") pod \"392681ce-fe14-4518-9697-efb94f82efdc\" (UID: \"392681ce-fe14-4518-9697-efb94f82efdc\") " Nov 02 01:49:26.159617 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:26.159599 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/392681ce-fe14-4518-9697-efb94f82efdc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/392681ce-fe14-4518-9697-efb94f82efdc-istio-podinfo\") pod \"392681ce-fe14-4518-9697-efb94f82efdc\" (UID: \"392681ce-fe14-4518-9697-efb94f82efdc\") " Nov 02 01:49:26.159746 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:26.159634 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/392681ce-fe14-4518-9697-efb94f82efdc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/392681ce-fe14-4518-9697-efb94f82efdc-istio-envoy\") pod \"392681ce-fe14-4518-9697-efb94f82efdc\" (UID: \"392681ce-fe14-4518-9697-efb94f82efdc\") " Nov 02 01:49:26.159746 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:26.159677 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/392681ce-fe14-4518-9697-efb94f82efdc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/392681ce-fe14-4518-9697-efb94f82efdc-istiod-ca-cert\") pod \"392681ce-fe14-4518-9697-efb94f82efdc\" (UID: \"392681ce-fe14-4518-9697-efb94f82efdc\") " Nov 02 01:49:26.159859 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:26.159744 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/392681ce-fe14-4518-9697-efb94f82efdc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/392681ce-fe14-4518-9697-efb94f82efdc-istio-data\") pod \"392681ce-fe14-4518-9697-efb94f82efdc\" (UID: \"392681ce-fe14-4518-9697-efb94f82efdc\") " Nov 02 01:49:26.159859 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:26.159805 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/392681ce-fe14-4518-9697-efb94f82efdc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/392681ce-fe14-4518-9697-efb94f82efdc-credential-socket\") pod \"392681ce-fe14-4518-9697-efb94f82efdc\" (UID: \"392681ce-fe14-4518-9697-efb94f82efdc\") " Nov 02 01:49:26.159962 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:26.159867 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/392681ce-fe14-4518-9697-efb94f82efdc-kube-api-access-4wld9\" (UniqueName: \"kubernetes.io/projected/392681ce-fe14-4518-9697-efb94f82efdc-kube-api-access-4wld9\") pod \"392681ce-fe14-4518-9697-efb94f82efdc\" (UID: \"392681ce-fe14-4518-9697-efb94f82efdc\") " Nov 02 01:49:26.159962 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:26.159870 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/392681ce-fe14-4518-9697-efb94f82efdc-workload-socket" pod "392681ce-fe14-4518-9697-efb94f82efdc" (UID: "392681ce-fe14-4518-9697-efb94f82efdc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:26.159962 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:26.159909 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/392681ce-fe14-4518-9697-efb94f82efdc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/392681ce-fe14-4518-9697-efb94f82efdc-workload-certs\") pod \"392681ce-fe14-4518-9697-efb94f82efdc\" (UID: \"392681ce-fe14-4518-9697-efb94f82efdc\") " Nov 02 01:49:26.160141 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:26.160113 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/392681ce-fe14-4518-9697-efb94f82efdc-istiod-ca-cert" pod "392681ce-fe14-4518-9697-efb94f82efdc" (UID: "392681ce-fe14-4518-9697-efb94f82efdc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:49:26.160484 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:26.160293 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/392681ce-fe14-4518-9697-efb94f82efdc-istio-data" pod "392681ce-fe14-4518-9697-efb94f82efdc" (UID: "392681ce-fe14-4518-9697-efb94f82efdc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:26.160484 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:26.160363 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/392681ce-fe14-4518-9697-efb94f82efdc-workload-certs" pod "392681ce-fe14-4518-9697-efb94f82efdc" (UID: "392681ce-fe14-4518-9697-efb94f82efdc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:26.160484 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:26.160449 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/392681ce-fe14-4518-9697-efb94f82efdc-credential-socket" pod "392681ce-fe14-4518-9697-efb94f82efdc" (UID: "392681ce-fe14-4518-9697-efb94f82efdc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:26.160835 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:26.160815 4427 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/392681ce-fe14-4518-9697-efb94f82efdc-istiod-ca-cert\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:26.160911 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:26.160847 4427 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/392681ce-fe14-4518-9697-efb94f82efdc-istio-data\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:26.160911 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:26.160864 4427 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/392681ce-fe14-4518-9697-efb94f82efdc-credential-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:26.160911 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:26.160879 4427 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/392681ce-fe14-4518-9697-efb94f82efdc-workload-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:26.160911 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:26.160894 4427 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/392681ce-fe14-4518-9697-efb94f82efdc-workload-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:26.162266 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:26.162242 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/392681ce-fe14-4518-9697-efb94f82efdc-istio-podinfo" pod "392681ce-fe14-4518-9697-efb94f82efdc" (UID: "392681ce-fe14-4518-9697-efb94f82efdc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:49:26.162463 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:26.162450 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/392681ce-fe14-4518-9697-efb94f82efdc-istio-envoy" pod "392681ce-fe14-4518-9697-efb94f82efdc" (UID: "392681ce-fe14-4518-9697-efb94f82efdc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:26.162542 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:26.162457 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/392681ce-fe14-4518-9697-efb94f82efdc-istio-token" pod "392681ce-fe14-4518-9697-efb94f82efdc" (UID: "392681ce-fe14-4518-9697-efb94f82efdc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:49:26.162542 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:26.162483 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/392681ce-fe14-4518-9697-efb94f82efdc-kube-api-access-4wld9" pod "392681ce-fe14-4518-9697-efb94f82efdc" (UID: "392681ce-fe14-4518-9697-efb94f82efdc"). InnerVolumeSpecName "kube-api-access-4wld9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:49:26.262391 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:26.262330 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wld9\" (UniqueName: \"kubernetes.io/projected/392681ce-fe14-4518-9697-efb94f82efdc-kube-api-access-4wld9\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:26.262391 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:26.262362 4427 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/392681ce-fe14-4518-9697-efb94f82efdc-istio-token\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:26.262646 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:26.262404 4427 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/392681ce-fe14-4518-9697-efb94f82efdc-istio-podinfo\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:26.262646 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:26.262419 4427 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/392681ce-fe14-4518-9697-efb94f82efdc-istio-envoy\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:26.316683 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:26.316646 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk"] Nov 02 01:49:26.322042 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:26.322012 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn"] Nov 02 01:49:26.334209 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:26.334182 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk"] Nov 02 01:49:26.483462 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:49:26.483358 4427 logging.go:55] [core] [Channel #3544 SubChannel #3545]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.149:50052", ServerName: "10.133.0.149:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.149:50052: operation was canceled" Nov 02 01:49:26.483462 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:26.483247 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d4a0-6595f44dff-m47jt" podUID="fca29702-c34c-4933-a93a-143c5481ba3a" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.149:50052\" within 1s: context deadline exceeded" Nov 02 01:49:27.077311 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:27.077269 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6" Nov 02 01:49:27.077311 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:27.077298 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6" event={"ID":"392681ce-fe14-4518-9697-efb94f82efdc","Type":"ContainerDied","Data":"58ba9384fe1bb0540213ab2d2acd40726951c3cbaf6f3f8e5f304923ae091032"} Nov 02 01:49:27.077601 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:27.077405 4427 scope.go:122] "RemoveContainer" containerID="44ced00972441350c17d5df842a755ef8ecae0ab1a661e0b090a4e0a376f9591" Nov 02 01:49:27.102782 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:27.102754 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6"] Nov 02 01:49:27.109844 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:27.109816 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0f32d6-openshift-default-5cc7c589c7-lqjc6"] Nov 02 01:49:27.304028 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:27.303986 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0f32d6-647fd89d8c-v79jr" podUID="17e05811-e6f0-4c5b-be43-d39429d1af11" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 01:49:27.533119 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:27.533081 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="392681ce-fe14-4518-9697-efb94f82efdc" path="/var/lib/kubelet/pods/392681ce-fe14-4518-9697-efb94f82efdc/volumes" Nov 02 01:49:27.533535 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:27.533486 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fca29702-c34c-4933-a93a-143c5481ba3a" path="/var/lib/kubelet/pods/fca29702-c34c-4933-a93a-143c5481ba3a/volumes" Nov 02 01:49:29.020756 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:29.020704 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv"] Nov 02 01:49:29.021323 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:29.020973 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv" podUID="d5cb7660-e2aa-48a2-9eac-508cd83f93cb" containerName="istio-proxy" containerID="cri-o://653247f1830371c147dfdeaf91a2a979e1c486b36c57a66759680b540f84909f" gracePeriod=30 Nov 02 01:49:29.027558 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:29.027526 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4"] Nov 02 01:49:29.027784 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:29.027756 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4" podUID="eb3258a9-adfb-46de-a0ab-7b4649948689" containerName="istio-proxy" containerID="cri-o://1b5572190e14ea57e5510ae221581192ac1496c471d71938270e59a7d347f04a" gracePeriod=30 Nov 02 01:49:31.582415 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:49:31.582131 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:49:32.863463 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:32.863424 4427 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 01:49:32.863938 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:32.863491 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4" podUID="eb3258a9-adfb-46de-a0ab-7b4649948689" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 01:49:32.978423 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:32.978362 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" podUID="9c056798-2cf8-47b1-9829-67cebda7a248" containerName="main" containerID="cri-o://f25cae758ce8cb349de3a1db3abe515c731d1858d620b5f27f21d0be55ee4a41" gracePeriod=45 Nov 02 01:49:33.123517 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:33.123432 4427 generic.go:356] "Generic (PLEG): container finished" podID="9c056798-2cf8-47b1-9829-67cebda7a248" containerID="f25cae758ce8cb349de3a1db3abe515c731d1858d620b5f27f21d0be55ee4a41" exitCode=0 Nov 02 01:49:33.123517 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:33.123495 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" event={"ID":"9c056798-2cf8-47b1-9829-67cebda7a248","Type":"ContainerDied","Data":"f25cae758ce8cb349de3a1db3abe515c731d1858d620b5f27f21d0be55ee4a41"} Nov 02 01:49:33.247345 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:33.247319 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" Nov 02 01:49:33.359368 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:33.359329 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c056798-2cf8-47b1-9829-67cebda7a248-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c056798-2cf8-47b1-9829-67cebda7a248-model-cache\") pod \"9c056798-2cf8-47b1-9829-67cebda7a248\" (UID: \"9c056798-2cf8-47b1-9829-67cebda7a248\") " Nov 02 01:49:33.359368 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:33.359402 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c056798-2cf8-47b1-9829-67cebda7a248-home\" (UniqueName: \"kubernetes.io/empty-dir/9c056798-2cf8-47b1-9829-67cebda7a248-home\") pod \"9c056798-2cf8-47b1-9829-67cebda7a248\" (UID: \"9c056798-2cf8-47b1-9829-67cebda7a248\") " Nov 02 01:49:33.359634 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:33.359455 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c056798-2cf8-47b1-9829-67cebda7a248-kube-api-access-57sbh\" (UniqueName: \"kubernetes.io/projected/9c056798-2cf8-47b1-9829-67cebda7a248-kube-api-access-57sbh\") pod \"9c056798-2cf8-47b1-9829-67cebda7a248\" (UID: \"9c056798-2cf8-47b1-9829-67cebda7a248\") " Nov 02 01:49:33.359634 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:33.359526 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c056798-2cf8-47b1-9829-67cebda7a248-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c056798-2cf8-47b1-9829-67cebda7a248-dshm\") pod \"9c056798-2cf8-47b1-9829-67cebda7a248\" (UID: \"9c056798-2cf8-47b1-9829-67cebda7a248\") " Nov 02 01:49:33.359634 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:33.359603 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c056798-2cf8-47b1-9829-67cebda7a248-tls-certs\" (UniqueName: \"kubernetes.io/secret/9c056798-2cf8-47b1-9829-67cebda7a248-tls-certs\") pod \"9c056798-2cf8-47b1-9829-67cebda7a248\" (UID: \"9c056798-2cf8-47b1-9829-67cebda7a248\") " Nov 02 01:49:33.359833 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:33.359635 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c056798-2cf8-47b1-9829-67cebda7a248-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c056798-2cf8-47b1-9829-67cebda7a248-tmp-dir\") pod \"9c056798-2cf8-47b1-9829-67cebda7a248\" (UID: \"9c056798-2cf8-47b1-9829-67cebda7a248\") " Nov 02 01:49:33.359833 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:33.359713 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c056798-2cf8-47b1-9829-67cebda7a248-home" pod "9c056798-2cf8-47b1-9829-67cebda7a248" (UID: "9c056798-2cf8-47b1-9829-67cebda7a248"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:33.360299 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:33.359954 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c056798-2cf8-47b1-9829-67cebda7a248-tmp-dir" pod "9c056798-2cf8-47b1-9829-67cebda7a248" (UID: "9c056798-2cf8-47b1-9829-67cebda7a248"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:33.360299 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:33.360160 4427 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c056798-2cf8-47b1-9829-67cebda7a248-tmp-dir\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:33.360299 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:33.360179 4427 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c056798-2cf8-47b1-9829-67cebda7a248-home\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:33.362469 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:33.362446 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c056798-2cf8-47b1-9829-67cebda7a248-dshm" pod "9c056798-2cf8-47b1-9829-67cebda7a248" (UID: "9c056798-2cf8-47b1-9829-67cebda7a248"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:33.362556 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:33.362529 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c056798-2cf8-47b1-9829-67cebda7a248-kube-api-access-57sbh" pod "9c056798-2cf8-47b1-9829-67cebda7a248" (UID: "9c056798-2cf8-47b1-9829-67cebda7a248"). InnerVolumeSpecName "kube-api-access-57sbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:49:33.362651 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:33.362631 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c056798-2cf8-47b1-9829-67cebda7a248-tls-certs" pod "9c056798-2cf8-47b1-9829-67cebda7a248" (UID: "9c056798-2cf8-47b1-9829-67cebda7a248"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:49:33.363489 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:33.363465 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c056798-2cf8-47b1-9829-67cebda7a248-model-cache" pod "9c056798-2cf8-47b1-9829-67cebda7a248" (UID: "9c056798-2cf8-47b1-9829-67cebda7a248"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:33.461506 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:33.461469 4427 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c056798-2cf8-47b1-9829-67cebda7a248-tls-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:33.461506 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:33.461501 4427 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c056798-2cf8-47b1-9829-67cebda7a248-model-cache\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:33.461506 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:33.461512 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57sbh\" (UniqueName: \"kubernetes.io/projected/9c056798-2cf8-47b1-9829-67cebda7a248-kube-api-access-57sbh\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:33.461749 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:33.461523 4427 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c056798-2cf8-47b1-9829-67cebda7a248-dshm\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:33.887787 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:33.887749 4427 patch_prober.go:29] interesting pod/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 01:49:33.888181 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:33.887812 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv" podUID="d5cb7660-e2aa-48a2-9eac-508cd83f93cb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 01:49:34.130935 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:34.130910 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" Nov 02 01:49:34.130935 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:34.130922 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr" event={"ID":"9c056798-2cf8-47b1-9829-67cebda7a248","Type":"ContainerDied","Data":"0f24700eda7f7fc576960de14a0fd9c7277d5b5eea444e28d5af54097e1f5775"} Nov 02 01:49:34.131186 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:34.130981 4427 scope.go:122] "RemoveContainer" containerID="f25cae758ce8cb349de3a1db3abe515c731d1858d620b5f27f21d0be55ee4a41" Nov 02 01:49:34.165438 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:34.165411 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr"] Nov 02 01:49:34.171550 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:34.171514 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ca1c53/distinct-w0-ca1c53-kserve-579fcb65df-j95qr"] Nov 02 01:49:34.290878 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:34.290851 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv" Nov 02 01:49:34.323874 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:34.323836 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4" Nov 02 01:49:34.373393 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:34.373338 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb3258a9-adfb-46de-a0ab-7b4649948689-istio-token\" (UniqueName: \"kubernetes.io/projected/eb3258a9-adfb-46de-a0ab-7b4649948689-istio-token\") pod \"eb3258a9-adfb-46de-a0ab-7b4649948689\" (UID: \"eb3258a9-adfb-46de-a0ab-7b4649948689\") " Nov 02 01:49:34.373607 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:34.373442 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb3258a9-adfb-46de-a0ab-7b4649948689-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb3258a9-adfb-46de-a0ab-7b4649948689-istio-envoy\") pod \"eb3258a9-adfb-46de-a0ab-7b4649948689\" (UID: \"eb3258a9-adfb-46de-a0ab-7b4649948689\") " Nov 02 01:49:34.373607 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:34.373483 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb3258a9-adfb-46de-a0ab-7b4649948689-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb3258a9-adfb-46de-a0ab-7b4649948689-istio-data\") pod \"eb3258a9-adfb-46de-a0ab-7b4649948689\" (UID: \"eb3258a9-adfb-46de-a0ab-7b4649948689\") " Nov 02 01:49:34.373607 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:34.373527 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5cb7660-e2aa-48a2-9eac-508cd83f93cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5cb7660-e2aa-48a2-9eac-508cd83f93cb-istio-envoy\") pod \"d5cb7660-e2aa-48a2-9eac-508cd83f93cb\" (UID: \"d5cb7660-e2aa-48a2-9eac-508cd83f93cb\") " Nov 02 01:49:34.373607 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:34.373558 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5cb7660-e2aa-48a2-9eac-508cd83f93cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5cb7660-e2aa-48a2-9eac-508cd83f93cb-workload-certs\") pod \"d5cb7660-e2aa-48a2-9eac-508cd83f93cb\" (UID: \"d5cb7660-e2aa-48a2-9eac-508cd83f93cb\") " Nov 02 01:49:34.373607 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:34.373588 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb3258a9-adfb-46de-a0ab-7b4649948689-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb3258a9-adfb-46de-a0ab-7b4649948689-istiod-ca-cert\") pod \"eb3258a9-adfb-46de-a0ab-7b4649948689\" (UID: \"eb3258a9-adfb-46de-a0ab-7b4649948689\") " Nov 02 01:49:34.373924 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:34.373673 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5cb7660-e2aa-48a2-9eac-508cd83f93cb-istio-token\" (UniqueName: \"kubernetes.io/projected/d5cb7660-e2aa-48a2-9eac-508cd83f93cb-istio-token\") pod \"d5cb7660-e2aa-48a2-9eac-508cd83f93cb\" (UID: \"d5cb7660-e2aa-48a2-9eac-508cd83f93cb\") " Nov 02 01:49:34.373924 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:34.373708 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5cb7660-e2aa-48a2-9eac-508cd83f93cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5cb7660-e2aa-48a2-9eac-508cd83f93cb-istio-podinfo\") pod \"d5cb7660-e2aa-48a2-9eac-508cd83f93cb\" (UID: \"d5cb7660-e2aa-48a2-9eac-508cd83f93cb\") " Nov 02 01:49:34.373924 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:34.373761 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5cb7660-e2aa-48a2-9eac-508cd83f93cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5cb7660-e2aa-48a2-9eac-508cd83f93cb-credential-socket\") pod \"d5cb7660-e2aa-48a2-9eac-508cd83f93cb\" (UID: \"d5cb7660-e2aa-48a2-9eac-508cd83f93cb\") " Nov 02 01:49:34.373924 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:34.373790 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5cb7660-e2aa-48a2-9eac-508cd83f93cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5cb7660-e2aa-48a2-9eac-508cd83f93cb-istio-data\") pod \"d5cb7660-e2aa-48a2-9eac-508cd83f93cb\" (UID: \"d5cb7660-e2aa-48a2-9eac-508cd83f93cb\") " Nov 02 01:49:34.373924 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:34.373837 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb3258a9-adfb-46de-a0ab-7b4649948689-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb3258a9-adfb-46de-a0ab-7b4649948689-credential-socket\") pod \"eb3258a9-adfb-46de-a0ab-7b4649948689\" (UID: \"eb3258a9-adfb-46de-a0ab-7b4649948689\") " Nov 02 01:49:34.373924 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:34.373868 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5cb7660-e2aa-48a2-9eac-508cd83f93cb-kube-api-access-hng4n\" (UniqueName: \"kubernetes.io/projected/d5cb7660-e2aa-48a2-9eac-508cd83f93cb-kube-api-access-hng4n\") pod \"d5cb7660-e2aa-48a2-9eac-508cd83f93cb\" (UID: \"d5cb7660-e2aa-48a2-9eac-508cd83f93cb\") " Nov 02 01:49:34.373924 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:34.373913 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb3258a9-adfb-46de-a0ab-7b4649948689-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb3258a9-adfb-46de-a0ab-7b4649948689-workload-socket\") pod \"eb3258a9-adfb-46de-a0ab-7b4649948689\" (UID: \"eb3258a9-adfb-46de-a0ab-7b4649948689\") " Nov 02 01:49:34.374345 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:34.373944 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb3258a9-adfb-46de-a0ab-7b4649948689-kube-api-access-s672s\" (UniqueName: \"kubernetes.io/projected/eb3258a9-adfb-46de-a0ab-7b4649948689-kube-api-access-s672s\") pod \"eb3258a9-adfb-46de-a0ab-7b4649948689\" (UID: \"eb3258a9-adfb-46de-a0ab-7b4649948689\") " Nov 02 01:49:34.374345 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:34.373993 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eb3258a9-adfb-46de-a0ab-7b4649948689-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb3258a9-adfb-46de-a0ab-7b4649948689-istio-podinfo\") pod \"eb3258a9-adfb-46de-a0ab-7b4649948689\" (UID: \"eb3258a9-adfb-46de-a0ab-7b4649948689\") " Nov 02 01:49:34.374345 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:34.374059 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5cb7660-e2aa-48a2-9eac-508cd83f93cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5cb7660-e2aa-48a2-9eac-508cd83f93cb-workload-socket\") pod \"d5cb7660-e2aa-48a2-9eac-508cd83f93cb\" (UID: \"d5cb7660-e2aa-48a2-9eac-508cd83f93cb\") " Nov 02 01:49:34.374345 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:34.374110 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5cb7660-e2aa-48a2-9eac-508cd83f93cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5cb7660-e2aa-48a2-9eac-508cd83f93cb-istiod-ca-cert\") pod \"d5cb7660-e2aa-48a2-9eac-508cd83f93cb\" (UID: \"d5cb7660-e2aa-48a2-9eac-508cd83f93cb\") " Nov 02 01:49:34.374345 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:34.374160 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb3258a9-adfb-46de-a0ab-7b4649948689-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb3258a9-adfb-46de-a0ab-7b4649948689-workload-certs\") pod \"eb3258a9-adfb-46de-a0ab-7b4649948689\" (UID: \"eb3258a9-adfb-46de-a0ab-7b4649948689\") " Nov 02 01:49:34.374902 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:34.374865 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb3258a9-adfb-46de-a0ab-7b4649948689-workload-certs" pod "eb3258a9-adfb-46de-a0ab-7b4649948689" (UID: "eb3258a9-adfb-46de-a0ab-7b4649948689"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:34.375080 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:34.374977 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb3258a9-adfb-46de-a0ab-7b4649948689-credential-socket" pod "eb3258a9-adfb-46de-a0ab-7b4649948689" (UID: "eb3258a9-adfb-46de-a0ab-7b4649948689"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:34.375080 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:34.375021 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb3258a9-adfb-46de-a0ab-7b4649948689-workload-socket" pod "eb3258a9-adfb-46de-a0ab-7b4649948689" (UID: "eb3258a9-adfb-46de-a0ab-7b4649948689"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:34.375405 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:34.375295 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb3258a9-adfb-46de-a0ab-7b4649948689-istio-data" pod "eb3258a9-adfb-46de-a0ab-7b4649948689" (UID: "eb3258a9-adfb-46de-a0ab-7b4649948689"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:34.375405 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:34.375306 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5cb7660-e2aa-48a2-9eac-508cd83f93cb-credential-socket" pod "d5cb7660-e2aa-48a2-9eac-508cd83f93cb" (UID: "d5cb7660-e2aa-48a2-9eac-508cd83f93cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:34.376675 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:34.375964 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5cb7660-e2aa-48a2-9eac-508cd83f93cb-istio-data" pod "d5cb7660-e2aa-48a2-9eac-508cd83f93cb" (UID: "d5cb7660-e2aa-48a2-9eac-508cd83f93cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:34.376675 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:34.376319 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb3258a9-adfb-46de-a0ab-7b4649948689-istiod-ca-cert" pod "eb3258a9-adfb-46de-a0ab-7b4649948689" (UID: "eb3258a9-adfb-46de-a0ab-7b4649948689"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:49:34.376675 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:34.376429 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5cb7660-e2aa-48a2-9eac-508cd83f93cb-workload-certs" pod "d5cb7660-e2aa-48a2-9eac-508cd83f93cb" (UID: "d5cb7660-e2aa-48a2-9eac-508cd83f93cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:34.377440 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:34.377405 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5cb7660-e2aa-48a2-9eac-508cd83f93cb-workload-socket" pod "d5cb7660-e2aa-48a2-9eac-508cd83f93cb" (UID: "d5cb7660-e2aa-48a2-9eac-508cd83f93cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:34.377681 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:34.377643 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5cb7660-e2aa-48a2-9eac-508cd83f93cb-istiod-ca-cert" pod "d5cb7660-e2aa-48a2-9eac-508cd83f93cb" (UID: "d5cb7660-e2aa-48a2-9eac-508cd83f93cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:49:34.383482 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:34.383442 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb3258a9-adfb-46de-a0ab-7b4649948689-kube-api-access-s672s" pod "eb3258a9-adfb-46de-a0ab-7b4649948689" (UID: "eb3258a9-adfb-46de-a0ab-7b4649948689"). InnerVolumeSpecName "kube-api-access-s672s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:49:34.383807 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:34.383781 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5cb7660-e2aa-48a2-9eac-508cd83f93cb-istio-envoy" pod "d5cb7660-e2aa-48a2-9eac-508cd83f93cb" (UID: "d5cb7660-e2aa-48a2-9eac-508cd83f93cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:34.384064 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:34.384036 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5cb7660-e2aa-48a2-9eac-508cd83f93cb-istio-token" pod "d5cb7660-e2aa-48a2-9eac-508cd83f93cb" (UID: "d5cb7660-e2aa-48a2-9eac-508cd83f93cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:49:34.384268 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:34.384227 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb3258a9-adfb-46de-a0ab-7b4649948689-istio-podinfo" pod "eb3258a9-adfb-46de-a0ab-7b4649948689" (UID: "eb3258a9-adfb-46de-a0ab-7b4649948689"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:49:34.384268 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:34.384256 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5cb7660-e2aa-48a2-9eac-508cd83f93cb-istio-podinfo" pod "d5cb7660-e2aa-48a2-9eac-508cd83f93cb" (UID: "d5cb7660-e2aa-48a2-9eac-508cd83f93cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:49:34.384560 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:34.384533 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb3258a9-adfb-46de-a0ab-7b4649948689-istio-envoy" pod "eb3258a9-adfb-46de-a0ab-7b4649948689" (UID: "eb3258a9-adfb-46de-a0ab-7b4649948689"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:34.384631 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:34.384607 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5cb7660-e2aa-48a2-9eac-508cd83f93cb-kube-api-access-hng4n" pod "d5cb7660-e2aa-48a2-9eac-508cd83f93cb" (UID: "d5cb7660-e2aa-48a2-9eac-508cd83f93cb"). InnerVolumeSpecName "kube-api-access-hng4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:49:34.384902 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:34.384875 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb3258a9-adfb-46de-a0ab-7b4649948689-istio-token" pod "eb3258a9-adfb-46de-a0ab-7b4649948689" (UID: "eb3258a9-adfb-46de-a0ab-7b4649948689"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:49:34.475940 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:34.475851 4427 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5cb7660-e2aa-48a2-9eac-508cd83f93cb-istio-token\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:34.475940 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:34.475883 4427 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5cb7660-e2aa-48a2-9eac-508cd83f93cb-istio-podinfo\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:34.475940 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:34.475899 4427 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5cb7660-e2aa-48a2-9eac-508cd83f93cb-credential-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:34.475940 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:34.475910 4427 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5cb7660-e2aa-48a2-9eac-508cd83f93cb-istio-data\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:34.475940 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:34.475919 4427 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb3258a9-adfb-46de-a0ab-7b4649948689-credential-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:34.475940 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:34.475928 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hng4n\" (UniqueName: \"kubernetes.io/projected/d5cb7660-e2aa-48a2-9eac-508cd83f93cb-kube-api-access-hng4n\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:34.475940 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:34.475937 4427 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb3258a9-adfb-46de-a0ab-7b4649948689-workload-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:34.475940 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:34.475946 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s672s\" (UniqueName: \"kubernetes.io/projected/eb3258a9-adfb-46de-a0ab-7b4649948689-kube-api-access-s672s\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:34.476317 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:34.475954 4427 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb3258a9-adfb-46de-a0ab-7b4649948689-istio-podinfo\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:34.476317 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:34.475963 4427 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5cb7660-e2aa-48a2-9eac-508cd83f93cb-workload-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:34.476317 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:34.475976 4427 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5cb7660-e2aa-48a2-9eac-508cd83f93cb-istiod-ca-cert\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:34.476317 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:34.475991 4427 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb3258a9-adfb-46de-a0ab-7b4649948689-workload-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:34.476317 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:34.475998 4427 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb3258a9-adfb-46de-a0ab-7b4649948689-istio-token\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:34.476317 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:34.476006 4427 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb3258a9-adfb-46de-a0ab-7b4649948689-istio-envoy\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:34.476317 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:34.476015 4427 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb3258a9-adfb-46de-a0ab-7b4649948689-istio-data\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:34.476317 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:34.476022 4427 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5cb7660-e2aa-48a2-9eac-508cd83f93cb-istio-envoy\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:34.476317 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:34.476030 4427 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5cb7660-e2aa-48a2-9eac-508cd83f93cb-workload-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:34.476317 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:34.476037 4427 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb3258a9-adfb-46de-a0ab-7b4649948689-istiod-ca-cert\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:35.138987 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:35.138948 4427 generic.go:356] "Generic (PLEG): container finished" podID="eb3258a9-adfb-46de-a0ab-7b4649948689" containerID="1b5572190e14ea57e5510ae221581192ac1496c471d71938270e59a7d347f04a" exitCode=0 Nov 02 01:49:35.139464 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:35.139003 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4" Nov 02 01:49:35.139464 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:35.139025 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4" event={"ID":"eb3258a9-adfb-46de-a0ab-7b4649948689","Type":"ContainerDied","Data":"1b5572190e14ea57e5510ae221581192ac1496c471d71938270e59a7d347f04a"} Nov 02 01:49:35.139464 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:35.139080 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4" event={"ID":"eb3258a9-adfb-46de-a0ab-7b4649948689","Type":"ContainerDied","Data":"3eef0c53616448963b113a190dbc03bea2dad6315d1fa9e9e09eba5bf68b3eaa"} Nov 02 01:49:35.139464 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:35.139101 4427 scope.go:122] "RemoveContainer" containerID="1b5572190e14ea57e5510ae221581192ac1496c471d71938270e59a7d347f04a" Nov 02 01:49:35.140462 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:35.140442 4427 generic.go:356] "Generic (PLEG): container finished" podID="d5cb7660-e2aa-48a2-9eac-508cd83f93cb" containerID="653247f1830371c147dfdeaf91a2a979e1c486b36c57a66759680b540f84909f" exitCode=0 Nov 02 01:49:35.140544 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:35.140503 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv" Nov 02 01:49:35.140544 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:35.140529 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv" event={"ID":"d5cb7660-e2aa-48a2-9eac-508cd83f93cb","Type":"ContainerDied","Data":"653247f1830371c147dfdeaf91a2a979e1c486b36c57a66759680b540f84909f"} Nov 02 01:49:35.140656 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:35.140584 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv" event={"ID":"d5cb7660-e2aa-48a2-9eac-508cd83f93cb","Type":"ContainerDied","Data":"bb08b8059619f95033814157e5ed08abc5b1b254aff8a0aa7f7bf814ce89b128"} Nov 02 01:49:35.151546 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:35.151526 4427 scope.go:122] "RemoveContainer" containerID="1b5572190e14ea57e5510ae221581192ac1496c471d71938270e59a7d347f04a" Nov 02 01:49:35.151866 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:49:35.151844 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b5572190e14ea57e5510ae221581192ac1496c471d71938270e59a7d347f04a\": container with ID starting with 1b5572190e14ea57e5510ae221581192ac1496c471d71938270e59a7d347f04a not found: ID does not exist" containerID="1b5572190e14ea57e5510ae221581192ac1496c471d71938270e59a7d347f04a" Nov 02 01:49:35.151960 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:35.151880 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b5572190e14ea57e5510ae221581192ac1496c471d71938270e59a7d347f04a"} err="failed to get container status \"1b5572190e14ea57e5510ae221581192ac1496c471d71938270e59a7d347f04a\": rpc error: code = NotFound desc = could not find container \"1b5572190e14ea57e5510ae221581192ac1496c471d71938270e59a7d347f04a\": container with ID starting with 1b5572190e14ea57e5510ae221581192ac1496c471d71938270e59a7d347f04a not found: ID does not exist" Nov 02 01:49:35.151960 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:35.151902 4427 scope.go:122] "RemoveContainer" containerID="653247f1830371c147dfdeaf91a2a979e1c486b36c57a66759680b540f84909f" Nov 02 01:49:35.164708 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:35.164684 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4"] Nov 02 01:49:35.170920 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:35.170895 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-n9qt4"] Nov 02 01:49:35.187116 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:35.187091 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv"] Nov 02 01:49:35.190368 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:35.190341 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5ed42b-openshift-default-d96fd7c5-q5mwv"] Nov 02 01:49:35.254613 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:35.254590 4427 scope.go:122] "RemoveContainer" containerID="653247f1830371c147dfdeaf91a2a979e1c486b36c57a66759680b540f84909f" Nov 02 01:49:35.255010 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:49:35.254986 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"653247f1830371c147dfdeaf91a2a979e1c486b36c57a66759680b540f84909f\": container with ID starting with 653247f1830371c147dfdeaf91a2a979e1c486b36c57a66759680b540f84909f not found: ID does not exist" containerID="653247f1830371c147dfdeaf91a2a979e1c486b36c57a66759680b540f84909f" Nov 02 01:49:35.255088 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:35.255021 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"653247f1830371c147dfdeaf91a2a979e1c486b36c57a66759680b540f84909f"} err="failed to get container status \"653247f1830371c147dfdeaf91a2a979e1c486b36c57a66759680b540f84909f\": rpc error: code = NotFound desc = could not find container \"653247f1830371c147dfdeaf91a2a979e1c486b36c57a66759680b540f84909f\": container with ID starting with 653247f1830371c147dfdeaf91a2a979e1c486b36c57a66759680b540f84909f not found: ID does not exist" Nov 02 01:49:35.403067 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:35.403037 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0f32d6-647fd89d8c-v79jr" Nov 02 01:49:35.486993 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:35.486949 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17e05811-e6f0-4c5b-be43-d39429d1af11-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17e05811-e6f0-4c5b-be43-d39429d1af11-plugins-config-volume\") pod \"17e05811-e6f0-4c5b-be43-d39429d1af11\" (UID: \"17e05811-e6f0-4c5b-be43-d39429d1af11\") " Nov 02 01:49:35.487194 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:35.487084 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17e05811-e6f0-4c5b-be43-d39429d1af11-kube-api-access-psqvd\" (UniqueName: \"kubernetes.io/projected/17e05811-e6f0-4c5b-be43-d39429d1af11-kube-api-access-psqvd\") pod \"17e05811-e6f0-4c5b-be43-d39429d1af11\" (UID: \"17e05811-e6f0-4c5b-be43-d39429d1af11\") " Nov 02 01:49:35.487341 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:35.487318 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17e05811-e6f0-4c5b-be43-d39429d1af11-plugins-config-volume" pod "17e05811-e6f0-4c5b-be43-d39429d1af11" (UID: "17e05811-e6f0-4c5b-be43-d39429d1af11"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:49:35.487515 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:35.487497 4427 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17e05811-e6f0-4c5b-be43-d39429d1af11-plugins-config-volume\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:35.489673 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:35.489641 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17e05811-e6f0-4c5b-be43-d39429d1af11-kube-api-access-psqvd" pod "17e05811-e6f0-4c5b-be43-d39429d1af11" (UID: "17e05811-e6f0-4c5b-be43-d39429d1af11"). InnerVolumeSpecName "kube-api-access-psqvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:49:35.533047 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:35.533013 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c056798-2cf8-47b1-9829-67cebda7a248" path="/var/lib/kubelet/pods/9c056798-2cf8-47b1-9829-67cebda7a248/volumes" Nov 02 01:49:35.533495 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:35.533480 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5cb7660-e2aa-48a2-9eac-508cd83f93cb" path="/var/lib/kubelet/pods/d5cb7660-e2aa-48a2-9eac-508cd83f93cb/volumes" Nov 02 01:49:35.533878 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:35.533866 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb3258a9-adfb-46de-a0ab-7b4649948689" path="/var/lib/kubelet/pods/eb3258a9-adfb-46de-a0ab-7b4649948689/volumes" Nov 02 01:49:35.588447 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:35.588350 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psqvd\" (UniqueName: \"kubernetes.io/projected/17e05811-e6f0-4c5b-be43-d39429d1af11-kube-api-access-psqvd\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:36.152436 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:36.152367 4427 generic.go:356] "Generic (PLEG): container finished" podID="17e05811-e6f0-4c5b-be43-d39429d1af11" containerID="4f20ca6d1ed8b5e95fc91ddac56434196c7a2daae0e290eb30bf9d6e6a38f59c" exitCode=0 Nov 02 01:49:36.152943 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:36.152463 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0f32d6-647fd89d8c-v79jr" Nov 02 01:49:36.152943 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:36.152549 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0f32d6-647fd89d8c-v79jr" event={"ID":"17e05811-e6f0-4c5b-be43-d39429d1af11","Type":"ContainerDied","Data":"4f20ca6d1ed8b5e95fc91ddac56434196c7a2daae0e290eb30bf9d6e6a38f59c"} Nov 02 01:49:36.152943 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:36.152605 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0f32d6-647fd89d8c-v79jr" event={"ID":"17e05811-e6f0-4c5b-be43-d39429d1af11","Type":"ContainerDied","Data":"f75840cf0220d6694f134f4e6a1758d000d9b05268887e4bdbf00253c2b78cab"} Nov 02 01:49:36.152943 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:36.152639 4427 scope.go:122] "RemoveContainer" containerID="4f20ca6d1ed8b5e95fc91ddac56434196c7a2daae0e290eb30bf9d6e6a38f59c" Nov 02 01:49:36.167467 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:36.167443 4427 scope.go:122] "RemoveContainer" containerID="4f20ca6d1ed8b5e95fc91ddac56434196c7a2daae0e290eb30bf9d6e6a38f59c" Nov 02 01:49:36.167815 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:49:36.167788 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f20ca6d1ed8b5e95fc91ddac56434196c7a2daae0e290eb30bf9d6e6a38f59c\": container with ID starting with 4f20ca6d1ed8b5e95fc91ddac56434196c7a2daae0e290eb30bf9d6e6a38f59c not found: ID does not exist" containerID="4f20ca6d1ed8b5e95fc91ddac56434196c7a2daae0e290eb30bf9d6e6a38f59c" Nov 02 01:49:36.167888 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:36.167827 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f20ca6d1ed8b5e95fc91ddac56434196c7a2daae0e290eb30bf9d6e6a38f59c"} err="failed to get container status \"4f20ca6d1ed8b5e95fc91ddac56434196c7a2daae0e290eb30bf9d6e6a38f59c\": rpc error: code = NotFound desc = could not find container \"4f20ca6d1ed8b5e95fc91ddac56434196c7a2daae0e290eb30bf9d6e6a38f59c\": container with ID starting with 4f20ca6d1ed8b5e95fc91ddac56434196c7a2daae0e290eb30bf9d6e6a38f59c not found: ID does not exist" Nov 02 01:49:36.232100 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:36.232060 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0f32d6-647fd89d8c-v79jr"] Nov 02 01:49:36.234266 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:36.234240 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ca1c53-cff6c4869-cmlfp"] Nov 02 01:49:36.234476 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:36.234456 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ca1c53-cff6c4869-cmlfp" podUID="fe3cde15-cd40-4b6a-9144-f96a05179a27" containerName="maas-api" containerID="cri-o://779cc2d78aedeb795aa0d2b16ed8694560b30d293757242756ab50e24d250587" gracePeriod=30 Nov 02 01:49:36.240282 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:36.240252 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0f32d6-647fd89d8c-v79jr"] Nov 02 01:49:36.510434 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:36.510407 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ca1c53-cff6c4869-cmlfp" Nov 02 01:49:36.599318 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:36.599281 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe3cde15-cd40-4b6a-9144-f96a05179a27-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe3cde15-cd40-4b6a-9144-f96a05179a27-maas-api-tls\") pod \"fe3cde15-cd40-4b6a-9144-f96a05179a27\" (UID: \"fe3cde15-cd40-4b6a-9144-f96a05179a27\") " Nov 02 01:49:36.599579 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:36.599327 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe3cde15-cd40-4b6a-9144-f96a05179a27-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe3cde15-cd40-4b6a-9144-f96a05179a27-openshift-service-ca\") pod \"fe3cde15-cd40-4b6a-9144-f96a05179a27\" (UID: \"fe3cde15-cd40-4b6a-9144-f96a05179a27\") " Nov 02 01:49:36.599579 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:36.599400 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe3cde15-cd40-4b6a-9144-f96a05179a27-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe3cde15-cd40-4b6a-9144-f96a05179a27-metrics-certs\") pod \"fe3cde15-cd40-4b6a-9144-f96a05179a27\" (UID: \"fe3cde15-cd40-4b6a-9144-f96a05179a27\") " Nov 02 01:49:36.599579 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:36.599429 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe3cde15-cd40-4b6a-9144-f96a05179a27-kube-api-access-fv58v\" (UniqueName: \"kubernetes.io/projected/fe3cde15-cd40-4b6a-9144-f96a05179a27-kube-api-access-fv58v\") pod \"fe3cde15-cd40-4b6a-9144-f96a05179a27\" (UID: \"fe3cde15-cd40-4b6a-9144-f96a05179a27\") " Nov 02 01:49:36.599809 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:36.599777 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe3cde15-cd40-4b6a-9144-f96a05179a27-openshift-service-ca" pod "fe3cde15-cd40-4b6a-9144-f96a05179a27" (UID: "fe3cde15-cd40-4b6a-9144-f96a05179a27"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:49:36.601777 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:36.601732 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe3cde15-cd40-4b6a-9144-f96a05179a27-metrics-certs" pod "fe3cde15-cd40-4b6a-9144-f96a05179a27" (UID: "fe3cde15-cd40-4b6a-9144-f96a05179a27"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:49:36.601979 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:36.601956 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe3cde15-cd40-4b6a-9144-f96a05179a27-maas-api-tls" pod "fe3cde15-cd40-4b6a-9144-f96a05179a27" (UID: "fe3cde15-cd40-4b6a-9144-f96a05179a27"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:49:36.602068 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:36.601958 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe3cde15-cd40-4b6a-9144-f96a05179a27-kube-api-access-fv58v" pod "fe3cde15-cd40-4b6a-9144-f96a05179a27" (UID: "fe3cde15-cd40-4b6a-9144-f96a05179a27"). InnerVolumeSpecName "kube-api-access-fv58v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:49:36.700981 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:36.700889 4427 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe3cde15-cd40-4b6a-9144-f96a05179a27-maas-api-tls\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:36.700981 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:36.700924 4427 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe3cde15-cd40-4b6a-9144-f96a05179a27-openshift-service-ca\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:36.700981 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:36.700938 4427 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe3cde15-cd40-4b6a-9144-f96a05179a27-metrics-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:36.700981 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:36.700950 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv58v\" (UniqueName: \"kubernetes.io/projected/fe3cde15-cd40-4b6a-9144-f96a05179a27-kube-api-access-fv58v\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:37.163019 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:37.162984 4427 generic.go:356] "Generic (PLEG): container finished" podID="fe3cde15-cd40-4b6a-9144-f96a05179a27" containerID="779cc2d78aedeb795aa0d2b16ed8694560b30d293757242756ab50e24d250587" exitCode=0 Nov 02 01:49:37.163535 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:37.163053 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ca1c53-cff6c4869-cmlfp" Nov 02 01:49:37.163535 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:37.163070 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ca1c53-cff6c4869-cmlfp" event={"ID":"fe3cde15-cd40-4b6a-9144-f96a05179a27","Type":"ContainerDied","Data":"779cc2d78aedeb795aa0d2b16ed8694560b30d293757242756ab50e24d250587"} Nov 02 01:49:37.163535 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:37.163139 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ca1c53-cff6c4869-cmlfp" event={"ID":"fe3cde15-cd40-4b6a-9144-f96a05179a27","Type":"ContainerDied","Data":"a2cda749dded8af96f29c25917936829587cd7c4dd2771479b657c5cc6a9a309"} Nov 02 01:49:37.163535 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:37.163172 4427 scope.go:122] "RemoveContainer" containerID="779cc2d78aedeb795aa0d2b16ed8694560b30d293757242756ab50e24d250587" Nov 02 01:49:37.173569 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:37.173542 4427 scope.go:122] "RemoveContainer" containerID="779cc2d78aedeb795aa0d2b16ed8694560b30d293757242756ab50e24d250587" Nov 02 01:49:37.173881 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:49:37.173861 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"779cc2d78aedeb795aa0d2b16ed8694560b30d293757242756ab50e24d250587\": container with ID starting with 779cc2d78aedeb795aa0d2b16ed8694560b30d293757242756ab50e24d250587 not found: ID does not exist" containerID="779cc2d78aedeb795aa0d2b16ed8694560b30d293757242756ab50e24d250587" Nov 02 01:49:37.173937 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:37.173895 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"779cc2d78aedeb795aa0d2b16ed8694560b30d293757242756ab50e24d250587"} err="failed to get container status \"779cc2d78aedeb795aa0d2b16ed8694560b30d293757242756ab50e24d250587\": rpc error: code = NotFound desc = could not find container \"779cc2d78aedeb795aa0d2b16ed8694560b30d293757242756ab50e24d250587\": container with ID starting with 779cc2d78aedeb795aa0d2b16ed8694560b30d293757242756ab50e24d250587 not found: ID does not exist" Nov 02 01:49:37.188674 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:37.188650 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ca1c53-cff6c4869-cmlfp"] Nov 02 01:49:37.195200 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:37.195173 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ca1c53-cff6c4869-cmlfp"] Nov 02 01:49:37.533906 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:37.533832 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17e05811-e6f0-4c5b-be43-d39429d1af11" path="/var/lib/kubelet/pods/17e05811-e6f0-4c5b-be43-d39429d1af11/volumes" Nov 02 01:49:37.534244 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:37.534231 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe3cde15-cd40-4b6a-9144-f96a05179a27" path="/var/lib/kubelet/pods/fe3cde15-cd40-4b6a-9144-f96a05179a27/volumes" Nov 02 01:49:40.669007 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:40.668972 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fm4lg"] Nov 02 01:49:40.669543 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:40.669229 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fm4lg" podUID="99cc3c61-f501-4a64-8f7e-0f41058aac68" containerName="istio-proxy" containerID="cri-o://c5e8b961667bc74f16c7e4da9d24aee04c4219311409da920d3a8a826adb3b87" gracePeriod=30 Nov 02 01:49:41.023550 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.023446 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93a027b3-m87mq"] Nov 02 01:49:41.032265 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.032225 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93a027b3-m87mq"] Nov 02 01:49:41.343282 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.343189 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" podUID="f67c954a-ce5b-41d9-8d99-7d126bd89d59" containerName="main" containerID="cri-o://1177e0754c41f78470f9ec6480093861574c18bb76394d04f56bae230a51ee03" gracePeriod=45 Nov 02 01:49:41.345173 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.345143 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" podUID="7085c0ab-7ee3-4665-abad-2977c25f68ba" containerName="main" containerID="cri-o://327e8fa337e2c1f15185a409ce92ec7095017b734e922c2f41d63467bf181484" gracePeriod=45 Nov 02 01:49:41.358847 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.358807 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" podUID="d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb" containerName="main" containerID="cri-o://625b727e7e64bfbe62ecb5c63f32f277e94d73e9a697a5f7dfae9d0227ffd763" gracePeriod=45 Nov 02 01:49:41.401734 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.401688 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv"] Nov 02 01:49:41.401943 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.401920 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv" podUID="819ae12d-e493-42d0-a549-811fb5fe7499" containerName="istio-proxy" containerID="cri-o://86bcb723f3ac09eb08da6a53144ab90378f17002034572135c475f435bf13c0c" gracePeriod=30 Nov 02 01:49:41.492346 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.484868 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" podUID="7085c0ab-7ee3-4665-abad-2977c25f68ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": dial tcp 10.133.0.45:8000: connect: connection refused" Nov 02 01:49:41.533487 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.533210 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f20bf903-25ba-4be2-8c17-2a10a0255e13" path="/var/lib/kubelet/pods/f20bf903-25ba-4be2-8c17-2a10a0255e13/volumes" Nov 02 01:49:41.647788 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.647756 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" Nov 02 01:49:41.689295 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.689256 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" Nov 02 01:49:41.699782 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.699732 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" Nov 02 01:49:41.756163 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.756126 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f67c954a-ce5b-41d9-8d99-7d126bd89d59-kube-api-access-pwh7v\" (UniqueName: \"kubernetes.io/projected/f67c954a-ce5b-41d9-8d99-7d126bd89d59-kube-api-access-pwh7v\") pod \"f67c954a-ce5b-41d9-8d99-7d126bd89d59\" (UID: \"f67c954a-ce5b-41d9-8d99-7d126bd89d59\") " Nov 02 01:49:41.756331 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.756259 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f67c954a-ce5b-41d9-8d99-7d126bd89d59-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f67c954a-ce5b-41d9-8d99-7d126bd89d59-tmp-dir\") pod \"f67c954a-ce5b-41d9-8d99-7d126bd89d59\" (UID: \"f67c954a-ce5b-41d9-8d99-7d126bd89d59\") " Nov 02 01:49:41.756331 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.756323 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f67c954a-ce5b-41d9-8d99-7d126bd89d59-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f67c954a-ce5b-41d9-8d99-7d126bd89d59-model-cache\") pod \"f67c954a-ce5b-41d9-8d99-7d126bd89d59\" (UID: \"f67c954a-ce5b-41d9-8d99-7d126bd89d59\") " Nov 02 01:49:41.756457 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.756397 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f67c954a-ce5b-41d9-8d99-7d126bd89d59-home\" (UniqueName: \"kubernetes.io/empty-dir/f67c954a-ce5b-41d9-8d99-7d126bd89d59-home\") pod \"f67c954a-ce5b-41d9-8d99-7d126bd89d59\" (UID: \"f67c954a-ce5b-41d9-8d99-7d126bd89d59\") " Nov 02 01:49:41.756504 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.756471 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f67c954a-ce5b-41d9-8d99-7d126bd89d59-tls-certs\" (UniqueName: \"kubernetes.io/secret/f67c954a-ce5b-41d9-8d99-7d126bd89d59-tls-certs\") pod \"f67c954a-ce5b-41d9-8d99-7d126bd89d59\" (UID: \"f67c954a-ce5b-41d9-8d99-7d126bd89d59\") " Nov 02 01:49:41.756541 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.756504 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f67c954a-ce5b-41d9-8d99-7d126bd89d59-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f67c954a-ce5b-41d9-8d99-7d126bd89d59-dshm\") pod \"f67c954a-ce5b-41d9-8d99-7d126bd89d59\" (UID: \"f67c954a-ce5b-41d9-8d99-7d126bd89d59\") " Nov 02 01:49:41.757576 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.757526 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f67c954a-ce5b-41d9-8d99-7d126bd89d59-model-cache" pod "f67c954a-ce5b-41d9-8d99-7d126bd89d59" (UID: "f67c954a-ce5b-41d9-8d99-7d126bd89d59"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:41.757744 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.757609 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f67c954a-ce5b-41d9-8d99-7d126bd89d59-home" pod "f67c954a-ce5b-41d9-8d99-7d126bd89d59" (UID: "f67c954a-ce5b-41d9-8d99-7d126bd89d59"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:41.757932 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.757899 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f67c954a-ce5b-41d9-8d99-7d126bd89d59-tmp-dir" pod "f67c954a-ce5b-41d9-8d99-7d126bd89d59" (UID: "f67c954a-ce5b-41d9-8d99-7d126bd89d59"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:41.760278 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.760249 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f67c954a-ce5b-41d9-8d99-7d126bd89d59-dshm" pod "f67c954a-ce5b-41d9-8d99-7d126bd89d59" (UID: "f67c954a-ce5b-41d9-8d99-7d126bd89d59"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:41.760760 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.760724 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f67c954a-ce5b-41d9-8d99-7d126bd89d59-tls-certs" pod "f67c954a-ce5b-41d9-8d99-7d126bd89d59" (UID: "f67c954a-ce5b-41d9-8d99-7d126bd89d59"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:49:41.760880 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.760851 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f67c954a-ce5b-41d9-8d99-7d126bd89d59-kube-api-access-pwh7v" pod "f67c954a-ce5b-41d9-8d99-7d126bd89d59" (UID: "f67c954a-ce5b-41d9-8d99-7d126bd89d59"). InnerVolumeSpecName "kube-api-access-pwh7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:49:41.857452 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.857409 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7085c0ab-7ee3-4665-abad-2977c25f68ba-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7085c0ab-7ee3-4665-abad-2977c25f68ba-tmp-dir\") pod \"7085c0ab-7ee3-4665-abad-2977c25f68ba\" (UID: \"7085c0ab-7ee3-4665-abad-2977c25f68ba\") " Nov 02 01:49:41.857668 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.857477 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb-home\" (UniqueName: \"kubernetes.io/empty-dir/d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb-home\") pod \"d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb\" (UID: \"d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb\") " Nov 02 01:49:41.857668 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.857541 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7085c0ab-7ee3-4665-abad-2977c25f68ba-tls-certs\" (UniqueName: \"kubernetes.io/secret/7085c0ab-7ee3-4665-abad-2977c25f68ba-tls-certs\") pod \"7085c0ab-7ee3-4665-abad-2977c25f68ba\" (UID: \"7085c0ab-7ee3-4665-abad-2977c25f68ba\") " Nov 02 01:49:41.857668 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.857571 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb-model-cache\") pod \"d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb\" (UID: \"d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb\") " Nov 02 01:49:41.857668 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.857601 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb-tmp-dir\") pod \"d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb\" (UID: \"d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb\") " Nov 02 01:49:41.857668 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.857640 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb-kube-api-access-lsbd7\" (UniqueName: \"kubernetes.io/projected/d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb-kube-api-access-lsbd7\") pod \"d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb\" (UID: \"d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb\") " Nov 02 01:49:41.857668 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.857666 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7085c0ab-7ee3-4665-abad-2977c25f68ba-kube-api-access-5bx4t\" (UniqueName: \"kubernetes.io/projected/7085c0ab-7ee3-4665-abad-2977c25f68ba-kube-api-access-5bx4t\") pod \"7085c0ab-7ee3-4665-abad-2977c25f68ba\" (UID: \"7085c0ab-7ee3-4665-abad-2977c25f68ba\") " Nov 02 01:49:41.857998 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.857700 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7085c0ab-7ee3-4665-abad-2977c25f68ba-home\" (UniqueName: \"kubernetes.io/empty-dir/7085c0ab-7ee3-4665-abad-2977c25f68ba-home\") pod \"7085c0ab-7ee3-4665-abad-2977c25f68ba\" (UID: \"7085c0ab-7ee3-4665-abad-2977c25f68ba\") " Nov 02 01:49:41.857998 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.857748 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb-dshm\") pod \"d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb\" (UID: \"d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb\") " Nov 02 01:49:41.857998 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.857773 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7085c0ab-7ee3-4665-abad-2977c25f68ba-tmp-dir" pod "7085c0ab-7ee3-4665-abad-2977c25f68ba" (UID: "7085c0ab-7ee3-4665-abad-2977c25f68ba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:41.857998 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.857784 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb-home" pod "d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb" (UID: "d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:41.857998 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.857785 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7085c0ab-7ee3-4665-abad-2977c25f68ba-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7085c0ab-7ee3-4665-abad-2977c25f68ba-dshm\") pod \"7085c0ab-7ee3-4665-abad-2977c25f68ba\" (UID: \"7085c0ab-7ee3-4665-abad-2977c25f68ba\") " Nov 02 01:49:41.857998 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.857862 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7085c0ab-7ee3-4665-abad-2977c25f68ba-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7085c0ab-7ee3-4665-abad-2977c25f68ba-model-cache\") pod \"7085c0ab-7ee3-4665-abad-2977c25f68ba\" (UID: \"7085c0ab-7ee3-4665-abad-2977c25f68ba\") " Nov 02 01:49:41.857998 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.857946 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb-tls-certs\" (UniqueName: \"kubernetes.io/secret/d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb-tls-certs\") pod \"d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb\" (UID: \"d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb\") " Nov 02 01:49:41.857998 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.857955 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb-tmp-dir" pod "d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb" (UID: "d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:41.858597 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.858009 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7085c0ab-7ee3-4665-abad-2977c25f68ba-home" pod "7085c0ab-7ee3-4665-abad-2977c25f68ba" (UID: "7085c0ab-7ee3-4665-abad-2977c25f68ba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:41.858597 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.858130 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb-model-cache" pod "d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb" (UID: "d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:41.858597 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.858136 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7085c0ab-7ee3-4665-abad-2977c25f68ba-model-cache" pod "7085c0ab-7ee3-4665-abad-2977c25f68ba" (UID: "7085c0ab-7ee3-4665-abad-2977c25f68ba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:41.858597 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.858421 4427 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f67c954a-ce5b-41d9-8d99-7d126bd89d59-home\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:41.858597 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.858443 4427 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f67c954a-ce5b-41d9-8d99-7d126bd89d59-tls-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:41.858597 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.858458 4427 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb-model-cache\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:41.858597 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.858497 4427 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb-tmp-dir\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:41.858597 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.858513 4427 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f67c954a-ce5b-41d9-8d99-7d126bd89d59-dshm\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:41.858597 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.858525 4427 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7085c0ab-7ee3-4665-abad-2977c25f68ba-home\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:41.858597 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.858541 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwh7v\" (UniqueName: \"kubernetes.io/projected/f67c954a-ce5b-41d9-8d99-7d126bd89d59-kube-api-access-pwh7v\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:41.858597 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.858555 4427 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7085c0ab-7ee3-4665-abad-2977c25f68ba-model-cache\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:41.858597 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.858569 4427 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f67c954a-ce5b-41d9-8d99-7d126bd89d59-tmp-dir\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:41.858597 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.858583 4427 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7085c0ab-7ee3-4665-abad-2977c25f68ba-tmp-dir\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:41.859279 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.858621 4427 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f67c954a-ce5b-41d9-8d99-7d126bd89d59-model-cache\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:41.859279 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.858731 4427 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb-home\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:41.860288 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.860251 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7085c0ab-7ee3-4665-abad-2977c25f68ba-dshm" pod "7085c0ab-7ee3-4665-abad-2977c25f68ba" (UID: "7085c0ab-7ee3-4665-abad-2977c25f68ba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:41.860423 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.860308 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb-kube-api-access-lsbd7" pod "d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb" (UID: "d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb"). InnerVolumeSpecName "kube-api-access-lsbd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:49:41.860579 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.860559 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7085c0ab-7ee3-4665-abad-2977c25f68ba-tls-certs" pod "7085c0ab-7ee3-4665-abad-2977c25f68ba" (UID: "7085c0ab-7ee3-4665-abad-2977c25f68ba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:49:41.860579 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.860568 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb-dshm" pod "d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb" (UID: "d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:41.860658 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.860630 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7085c0ab-7ee3-4665-abad-2977c25f68ba-kube-api-access-5bx4t" pod "7085c0ab-7ee3-4665-abad-2977c25f68ba" (UID: "7085c0ab-7ee3-4665-abad-2977c25f68ba"). InnerVolumeSpecName "kube-api-access-5bx4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:49:41.860748 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.860735 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb-tls-certs" pod "d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb" (UID: "d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:49:41.959993 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.959917 4427 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7085c0ab-7ee3-4665-abad-2977c25f68ba-tls-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:41.959993 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.959948 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsbd7\" (UniqueName: \"kubernetes.io/projected/d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb-kube-api-access-lsbd7\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:41.959993 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.959960 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bx4t\" (UniqueName: \"kubernetes.io/projected/7085c0ab-7ee3-4665-abad-2977c25f68ba-kube-api-access-5bx4t\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:41.959993 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.959969 4427 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb-dshm\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:41.959993 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.959978 4427 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7085c0ab-7ee3-4665-abad-2977c25f68ba-dshm\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:41.959993 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:41.959986 4427 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb-tls-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:42.198007 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:42.197966 4427 generic.go:356] "Generic (PLEG): container finished" podID="d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb" containerID="625b727e7e64bfbe62ecb5c63f32f277e94d73e9a697a5f7dfae9d0227ffd763" exitCode=0 Nov 02 01:49:42.198218 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:42.198016 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" Nov 02 01:49:42.198218 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:42.198040 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" event={"ID":"d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb","Type":"ContainerDied","Data":"625b727e7e64bfbe62ecb5c63f32f277e94d73e9a697a5f7dfae9d0227ffd763"} Nov 02 01:49:42.198218 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:42.198089 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk" event={"ID":"d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb","Type":"ContainerDied","Data":"cbe1229b744d9aeac6e0cdef8ab2da1bc540bf75d7c37bcbd5008fb295a3a66d"} Nov 02 01:49:42.198218 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:42.198111 4427 scope.go:122] "RemoveContainer" containerID="625b727e7e64bfbe62ecb5c63f32f277e94d73e9a697a5f7dfae9d0227ffd763" Nov 02 01:49:42.199555 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:42.199534 4427 generic.go:356] "Generic (PLEG): container finished" podID="7085c0ab-7ee3-4665-abad-2977c25f68ba" containerID="327e8fa337e2c1f15185a409ce92ec7095017b734e922c2f41d63467bf181484" exitCode=0 Nov 02 01:49:42.199656 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:42.199581 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" Nov 02 01:49:42.199715 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:42.199662 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" event={"ID":"7085c0ab-7ee3-4665-abad-2977c25f68ba","Type":"ContainerDied","Data":"327e8fa337e2c1f15185a409ce92ec7095017b734e922c2f41d63467bf181484"} Nov 02 01:49:42.199715 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:42.199695 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn" event={"ID":"7085c0ab-7ee3-4665-abad-2977c25f68ba","Type":"ContainerDied","Data":"5ef37606d1bbc9aa2d4505b6487f53dc4f28893ee0039634d3efb2a4a5ff4e79"} Nov 02 01:49:42.201254 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:42.201232 4427 generic.go:356] "Generic (PLEG): container finished" podID="f67c954a-ce5b-41d9-8d99-7d126bd89d59" containerID="1177e0754c41f78470f9ec6480093861574c18bb76394d04f56bae230a51ee03" exitCode=0 Nov 02 01:49:42.201343 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:42.201263 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" event={"ID":"f67c954a-ce5b-41d9-8d99-7d126bd89d59","Type":"ContainerDied","Data":"1177e0754c41f78470f9ec6480093861574c18bb76394d04f56bae230a51ee03"} Nov 02 01:49:42.201343 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:42.201302 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" event={"ID":"f67c954a-ce5b-41d9-8d99-7d126bd89d59","Type":"ContainerDied","Data":"f4141605995397b676fb7e47daec7f28e443b6eb9fbc6b65ac2cdf652e0b31df"} Nov 02 01:49:42.201343 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:42.201302 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk" Nov 02 01:49:42.215042 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:42.215021 4427 scope.go:122] "RemoveContainer" containerID="625b727e7e64bfbe62ecb5c63f32f277e94d73e9a697a5f7dfae9d0227ffd763" Nov 02 01:49:42.215569 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:49:42.215542 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"625b727e7e64bfbe62ecb5c63f32f277e94d73e9a697a5f7dfae9d0227ffd763\": container with ID starting with 625b727e7e64bfbe62ecb5c63f32f277e94d73e9a697a5f7dfae9d0227ffd763 not found: ID does not exist" containerID="625b727e7e64bfbe62ecb5c63f32f277e94d73e9a697a5f7dfae9d0227ffd763" Nov 02 01:49:42.215670 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:42.215584 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"625b727e7e64bfbe62ecb5c63f32f277e94d73e9a697a5f7dfae9d0227ffd763"} err="failed to get container status \"625b727e7e64bfbe62ecb5c63f32f277e94d73e9a697a5f7dfae9d0227ffd763\": rpc error: code = NotFound desc = could not find container \"625b727e7e64bfbe62ecb5c63f32f277e94d73e9a697a5f7dfae9d0227ffd763\": container with ID starting with 625b727e7e64bfbe62ecb5c63f32f277e94d73e9a697a5f7dfae9d0227ffd763 not found: ID does not exist" Nov 02 01:49:42.215670 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:42.215603 4427 scope.go:122] "RemoveContainer" containerID="327e8fa337e2c1f15185a409ce92ec7095017b734e922c2f41d63467bf181484" Nov 02 01:49:42.224298 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:42.224278 4427 scope.go:122] "RemoveContainer" containerID="327e8fa337e2c1f15185a409ce92ec7095017b734e922c2f41d63467bf181484" Nov 02 01:49:42.224616 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:49:42.224597 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"327e8fa337e2c1f15185a409ce92ec7095017b734e922c2f41d63467bf181484\": container with ID starting with 327e8fa337e2c1f15185a409ce92ec7095017b734e922c2f41d63467bf181484 not found: ID does not exist" containerID="327e8fa337e2c1f15185a409ce92ec7095017b734e922c2f41d63467bf181484" Nov 02 01:49:42.224671 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:42.224626 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"327e8fa337e2c1f15185a409ce92ec7095017b734e922c2f41d63467bf181484"} err="failed to get container status \"327e8fa337e2c1f15185a409ce92ec7095017b734e922c2f41d63467bf181484\": rpc error: code = NotFound desc = could not find container \"327e8fa337e2c1f15185a409ce92ec7095017b734e922c2f41d63467bf181484\": container with ID starting with 327e8fa337e2c1f15185a409ce92ec7095017b734e922c2f41d63467bf181484 not found: ID does not exist" Nov 02 01:49:42.224671 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:42.224643 4427 scope.go:122] "RemoveContainer" containerID="1177e0754c41f78470f9ec6480093861574c18bb76394d04f56bae230a51ee03" Nov 02 01:49:42.228444 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:42.228420 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk"] Nov 02 01:49:42.232667 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:42.232632 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-123986/simulator-w5-123986-kserve-65dfb74c9-k5tzk"] Nov 02 01:49:42.233833 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:42.233814 4427 scope.go:122] "RemoveContainer" containerID="1177e0754c41f78470f9ec6480093861574c18bb76394d04f56bae230a51ee03" Nov 02 01:49:42.234142 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:49:42.234121 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1177e0754c41f78470f9ec6480093861574c18bb76394d04f56bae230a51ee03\": container with ID starting with 1177e0754c41f78470f9ec6480093861574c18bb76394d04f56bae230a51ee03 not found: ID does not exist" containerID="1177e0754c41f78470f9ec6480093861574c18bb76394d04f56bae230a51ee03" Nov 02 01:49:42.234188 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:42.234149 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1177e0754c41f78470f9ec6480093861574c18bb76394d04f56bae230a51ee03"} err="failed to get container status \"1177e0754c41f78470f9ec6480093861574c18bb76394d04f56bae230a51ee03\": rpc error: code = NotFound desc = could not find container \"1177e0754c41f78470f9ec6480093861574c18bb76394d04f56bae230a51ee03\": container with ID starting with 1177e0754c41f78470f9ec6480093861574c18bb76394d04f56bae230a51ee03 not found: ID does not exist" Nov 02 01:49:42.245694 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:42.245662 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn"] Nov 02 01:49:42.250841 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:42.250817 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-123986/premium-w5-123986-kserve-6d567889d6-fc7cn"] Nov 02 01:49:42.263480 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:42.263454 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk"] Nov 02 01:49:42.267194 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:42.267170 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-123986/distinct-w5-123986-kserve-79b985685c-xvlxk"] Nov 02 01:49:43.532157 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:43.532124 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7085c0ab-7ee3-4665-abad-2977c25f68ba" path="/var/lib/kubelet/pods/7085c0ab-7ee3-4665-abad-2977c25f68ba/volumes" Nov 02 01:49:43.532717 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:43.532628 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb" path="/var/lib/kubelet/pods/d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb/volumes" Nov 02 01:49:43.532992 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:43.532978 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f67c954a-ce5b-41d9-8d99-7d126bd89d59" path="/var/lib/kubelet/pods/f67c954a-ce5b-41d9-8d99-7d126bd89d59/volumes" Nov 02 01:49:43.692398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:43.691583 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c"] Nov 02 01:49:43.692398 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:43.691832 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c" podUID="c37ba466-a830-4246-832b-a8a4d0b5ab9f" containerName="istio-proxy" containerID="cri-o://73d46afa0f9cf0460c0eb36b5ffc4af31c63614272d021f283b01568394d56f8" gracePeriod=30 Nov 02 01:49:45.935663 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:45.935632 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fm4lg" Nov 02 01:49:46.110713 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.110679 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99cc3c61-f501-4a64-8f7e-0f41058aac68-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99cc3c61-f501-4a64-8f7e-0f41058aac68-istio-envoy\") pod \"99cc3c61-f501-4a64-8f7e-0f41058aac68\" (UID: \"99cc3c61-f501-4a64-8f7e-0f41058aac68\") " Nov 02 01:49:46.110913 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.110724 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99cc3c61-f501-4a64-8f7e-0f41058aac68-istio-token\" (UniqueName: \"kubernetes.io/projected/99cc3c61-f501-4a64-8f7e-0f41058aac68-istio-token\") pod \"99cc3c61-f501-4a64-8f7e-0f41058aac68\" (UID: \"99cc3c61-f501-4a64-8f7e-0f41058aac68\") " Nov 02 01:49:46.110913 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.110789 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99cc3c61-f501-4a64-8f7e-0f41058aac68-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99cc3c61-f501-4a64-8f7e-0f41058aac68-credential-socket\") pod \"99cc3c61-f501-4a64-8f7e-0f41058aac68\" (UID: \"99cc3c61-f501-4a64-8f7e-0f41058aac68\") " Nov 02 01:49:46.110913 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.110810 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99cc3c61-f501-4a64-8f7e-0f41058aac68-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99cc3c61-f501-4a64-8f7e-0f41058aac68-workload-certs\") pod \"99cc3c61-f501-4a64-8f7e-0f41058aac68\" (UID: \"99cc3c61-f501-4a64-8f7e-0f41058aac68\") " Nov 02 01:49:46.110913 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.110827 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99cc3c61-f501-4a64-8f7e-0f41058aac68-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99cc3c61-f501-4a64-8f7e-0f41058aac68-workload-socket\") pod \"99cc3c61-f501-4a64-8f7e-0f41058aac68\" (UID: \"99cc3c61-f501-4a64-8f7e-0f41058aac68\") " Nov 02 01:49:46.110913 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.110862 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99cc3c61-f501-4a64-8f7e-0f41058aac68-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99cc3c61-f501-4a64-8f7e-0f41058aac68-istiod-ca-cert\") pod \"99cc3c61-f501-4a64-8f7e-0f41058aac68\" (UID: \"99cc3c61-f501-4a64-8f7e-0f41058aac68\") " Nov 02 01:49:46.111203 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.110915 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99cc3c61-f501-4a64-8f7e-0f41058aac68-kube-api-access-5rw4m\" (UniqueName: \"kubernetes.io/projected/99cc3c61-f501-4a64-8f7e-0f41058aac68-kube-api-access-5rw4m\") pod \"99cc3c61-f501-4a64-8f7e-0f41058aac68\" (UID: \"99cc3c61-f501-4a64-8f7e-0f41058aac68\") " Nov 02 01:49:46.111203 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.110989 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/99cc3c61-f501-4a64-8f7e-0f41058aac68-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99cc3c61-f501-4a64-8f7e-0f41058aac68-istio-podinfo\") pod \"99cc3c61-f501-4a64-8f7e-0f41058aac68\" (UID: \"99cc3c61-f501-4a64-8f7e-0f41058aac68\") " Nov 02 01:49:46.111203 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.111047 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99cc3c61-f501-4a64-8f7e-0f41058aac68-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99cc3c61-f501-4a64-8f7e-0f41058aac68-istio-data\") pod \"99cc3c61-f501-4a64-8f7e-0f41058aac68\" (UID: \"99cc3c61-f501-4a64-8f7e-0f41058aac68\") " Nov 02 01:49:46.111203 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.111133 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99cc3c61-f501-4a64-8f7e-0f41058aac68-credential-socket" pod "99cc3c61-f501-4a64-8f7e-0f41058aac68" (UID: "99cc3c61-f501-4a64-8f7e-0f41058aac68"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:46.111470 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.111204 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99cc3c61-f501-4a64-8f7e-0f41058aac68-workload-certs" pod "99cc3c61-f501-4a64-8f7e-0f41058aac68" (UID: "99cc3c61-f501-4a64-8f7e-0f41058aac68"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:46.111470 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.111448 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99cc3c61-f501-4a64-8f7e-0f41058aac68-istio-data" pod "99cc3c61-f501-4a64-8f7e-0f41058aac68" (UID: "99cc3c61-f501-4a64-8f7e-0f41058aac68"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:46.111594 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.111572 4427 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99cc3c61-f501-4a64-8f7e-0f41058aac68-credential-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:46.111658 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.111597 4427 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99cc3c61-f501-4a64-8f7e-0f41058aac68-workload-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:46.111658 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.111611 4427 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99cc3c61-f501-4a64-8f7e-0f41058aac68-istio-data\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:46.111658 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.111620 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99cc3c61-f501-4a64-8f7e-0f41058aac68-istiod-ca-cert" pod "99cc3c61-f501-4a64-8f7e-0f41058aac68" (UID: "99cc3c61-f501-4a64-8f7e-0f41058aac68"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:49:46.111764 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.111597 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99cc3c61-f501-4a64-8f7e-0f41058aac68-workload-socket" pod "99cc3c61-f501-4a64-8f7e-0f41058aac68" (UID: "99cc3c61-f501-4a64-8f7e-0f41058aac68"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:46.113321 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.113293 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99cc3c61-f501-4a64-8f7e-0f41058aac68-istio-envoy" pod "99cc3c61-f501-4a64-8f7e-0f41058aac68" (UID: "99cc3c61-f501-4a64-8f7e-0f41058aac68"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:46.113321 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.113306 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99cc3c61-f501-4a64-8f7e-0f41058aac68-istio-token" pod "99cc3c61-f501-4a64-8f7e-0f41058aac68" (UID: "99cc3c61-f501-4a64-8f7e-0f41058aac68"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:49:46.113522 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.113431 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99cc3c61-f501-4a64-8f7e-0f41058aac68-kube-api-access-5rw4m" pod "99cc3c61-f501-4a64-8f7e-0f41058aac68" (UID: "99cc3c61-f501-4a64-8f7e-0f41058aac68"). InnerVolumeSpecName "kube-api-access-5rw4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:49:46.113599 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.113571 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/99cc3c61-f501-4a64-8f7e-0f41058aac68-istio-podinfo" pod "99cc3c61-f501-4a64-8f7e-0f41058aac68" (UID: "99cc3c61-f501-4a64-8f7e-0f41058aac68"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:49:46.212527 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.212493 4427 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99cc3c61-f501-4a64-8f7e-0f41058aac68-workload-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:46.212527 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.212523 4427 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99cc3c61-f501-4a64-8f7e-0f41058aac68-istiod-ca-cert\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:46.212527 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.212533 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rw4m\" (UniqueName: \"kubernetes.io/projected/99cc3c61-f501-4a64-8f7e-0f41058aac68-kube-api-access-5rw4m\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:46.212767 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.212544 4427 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99cc3c61-f501-4a64-8f7e-0f41058aac68-istio-podinfo\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:46.212767 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.212553 4427 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99cc3c61-f501-4a64-8f7e-0f41058aac68-istio-envoy\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:46.212767 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.212561 4427 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99cc3c61-f501-4a64-8f7e-0f41058aac68-istio-token\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:46.227594 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.227568 4427 generic.go:356] "Generic (PLEG): container finished" podID="99cc3c61-f501-4a64-8f7e-0f41058aac68" containerID="c5e8b961667bc74f16c7e4da9d24aee04c4219311409da920d3a8a826adb3b87" exitCode=0 Nov 02 01:49:46.227738 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.227616 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fm4lg" Nov 02 01:49:46.227738 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.227640 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fm4lg" event={"ID":"99cc3c61-f501-4a64-8f7e-0f41058aac68","Type":"ContainerDied","Data":"c5e8b961667bc74f16c7e4da9d24aee04c4219311409da920d3a8a826adb3b87"} Nov 02 01:49:46.227738 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.227684 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fm4lg" event={"ID":"99cc3c61-f501-4a64-8f7e-0f41058aac68","Type":"ContainerDied","Data":"b0bc9d45d0aa63516c236033d95d5c596e21f4b0517626dc2744b48eb77858f3"} Nov 02 01:49:46.227738 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.227704 4427 scope.go:122] "RemoveContainer" containerID="c5e8b961667bc74f16c7e4da9d24aee04c4219311409da920d3a8a826adb3b87" Nov 02 01:49:46.240603 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.240579 4427 scope.go:122] "RemoveContainer" containerID="c5e8b961667bc74f16c7e4da9d24aee04c4219311409da920d3a8a826adb3b87" Nov 02 01:49:46.240925 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:49:46.240899 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5e8b961667bc74f16c7e4da9d24aee04c4219311409da920d3a8a826adb3b87\": container with ID starting with c5e8b961667bc74f16c7e4da9d24aee04c4219311409da920d3a8a826adb3b87 not found: ID does not exist" containerID="c5e8b961667bc74f16c7e4da9d24aee04c4219311409da920d3a8a826adb3b87" Nov 02 01:49:46.241029 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.240930 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5e8b961667bc74f16c7e4da9d24aee04c4219311409da920d3a8a826adb3b87"} err="failed to get container status \"c5e8b961667bc74f16c7e4da9d24aee04c4219311409da920d3a8a826adb3b87\": rpc error: code = NotFound desc = could not find container \"c5e8b961667bc74f16c7e4da9d24aee04c4219311409da920d3a8a826adb3b87\": container with ID starting with c5e8b961667bc74f16c7e4da9d24aee04c4219311409da920d3a8a826adb3b87 not found: ID does not exist" Nov 02 01:49:46.256892 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.256862 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fm4lg"] Nov 02 01:49:46.263326 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.263300 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-fm4lg"] Nov 02 01:49:46.666677 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.666604 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv" Nov 02 01:49:46.798741 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.798706 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-123986-86c99dc5d5-7n5p8"] Nov 02 01:49:46.798988 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.798961 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-123986-86c99dc5d5-7n5p8" podUID="f12da917-8a8f-445b-b939-0bab19110258" containerName="maas-api" containerID="cri-o://e7b24774ee2a5b38c99290040b1d604c87842dd18b5dc58f99e340169859ce05" gracePeriod=30 Nov 02 01:49:46.821908 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.821873 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/819ae12d-e493-42d0-a549-811fb5fe7499-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/819ae12d-e493-42d0-a549-811fb5fe7499-istiod-ca-cert\") pod \"819ae12d-e493-42d0-a549-811fb5fe7499\" (UID: \"819ae12d-e493-42d0-a549-811fb5fe7499\") " Nov 02 01:49:46.822065 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.821945 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/819ae12d-e493-42d0-a549-811fb5fe7499-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/819ae12d-e493-42d0-a549-811fb5fe7499-credential-socket\") pod \"819ae12d-e493-42d0-a549-811fb5fe7499\" (UID: \"819ae12d-e493-42d0-a549-811fb5fe7499\") " Nov 02 01:49:46.822065 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.822001 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/819ae12d-e493-42d0-a549-811fb5fe7499-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/819ae12d-e493-42d0-a549-811fb5fe7499-workload-certs\") pod \"819ae12d-e493-42d0-a549-811fb5fe7499\" (UID: \"819ae12d-e493-42d0-a549-811fb5fe7499\") " Nov 02 01:49:46.822065 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.822053 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/819ae12d-e493-42d0-a549-811fb5fe7499-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/819ae12d-e493-42d0-a549-811fb5fe7499-istio-podinfo\") pod \"819ae12d-e493-42d0-a549-811fb5fe7499\" (UID: \"819ae12d-e493-42d0-a549-811fb5fe7499\") " Nov 02 01:49:46.822264 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.822085 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/819ae12d-e493-42d0-a549-811fb5fe7499-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/819ae12d-e493-42d0-a549-811fb5fe7499-workload-socket\") pod \"819ae12d-e493-42d0-a549-811fb5fe7499\" (UID: \"819ae12d-e493-42d0-a549-811fb5fe7499\") " Nov 02 01:49:46.822264 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.822162 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/819ae12d-e493-42d0-a549-811fb5fe7499-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/819ae12d-e493-42d0-a549-811fb5fe7499-istio-data\") pod \"819ae12d-e493-42d0-a549-811fb5fe7499\" (UID: \"819ae12d-e493-42d0-a549-811fb5fe7499\") " Nov 02 01:49:46.822264 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.822214 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/819ae12d-e493-42d0-a549-811fb5fe7499-kube-api-access-6jh6k\" (UniqueName: \"kubernetes.io/projected/819ae12d-e493-42d0-a549-811fb5fe7499-kube-api-access-6jh6k\") pod \"819ae12d-e493-42d0-a549-811fb5fe7499\" (UID: \"819ae12d-e493-42d0-a549-811fb5fe7499\") " Nov 02 01:49:46.822544 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.822311 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/819ae12d-e493-42d0-a549-811fb5fe7499-istio-token\" (UniqueName: \"kubernetes.io/projected/819ae12d-e493-42d0-a549-811fb5fe7499-istio-token\") pod \"819ae12d-e493-42d0-a549-811fb5fe7499\" (UID: \"819ae12d-e493-42d0-a549-811fb5fe7499\") " Nov 02 01:49:46.822544 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.822363 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/819ae12d-e493-42d0-a549-811fb5fe7499-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/819ae12d-e493-42d0-a549-811fb5fe7499-istio-envoy\") pod \"819ae12d-e493-42d0-a549-811fb5fe7499\" (UID: \"819ae12d-e493-42d0-a549-811fb5fe7499\") " Nov 02 01:49:46.823394 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.823349 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/819ae12d-e493-42d0-a549-811fb5fe7499-workload-certs" pod "819ae12d-e493-42d0-a549-811fb5fe7499" (UID: "819ae12d-e493-42d0-a549-811fb5fe7499"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:46.831131 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.824659 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/819ae12d-e493-42d0-a549-811fb5fe7499-istio-data" pod "819ae12d-e493-42d0-a549-811fb5fe7499" (UID: "819ae12d-e493-42d0-a549-811fb5fe7499"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:46.831131 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.828313 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/819ae12d-e493-42d0-a549-811fb5fe7499-istiod-ca-cert" pod "819ae12d-e493-42d0-a549-811fb5fe7499" (UID: "819ae12d-e493-42d0-a549-811fb5fe7499"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:49:46.831131 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.828600 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/819ae12d-e493-42d0-a549-811fb5fe7499-credential-socket" pod "819ae12d-e493-42d0-a549-811fb5fe7499" (UID: "819ae12d-e493-42d0-a549-811fb5fe7499"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:46.831131 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.828797 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/819ae12d-e493-42d0-a549-811fb5fe7499-workload-socket" pod "819ae12d-e493-42d0-a549-811fb5fe7499" (UID: "819ae12d-e493-42d0-a549-811fb5fe7499"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:46.834085 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.834048 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/819ae12d-e493-42d0-a549-811fb5fe7499-istio-podinfo" pod "819ae12d-e493-42d0-a549-811fb5fe7499" (UID: "819ae12d-e493-42d0-a549-811fb5fe7499"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:49:46.837940 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.837907 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/819ae12d-e493-42d0-a549-811fb5fe7499-istio-token" pod "819ae12d-e493-42d0-a549-811fb5fe7499" (UID: "819ae12d-e493-42d0-a549-811fb5fe7499"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:49:46.838070 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.838017 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/819ae12d-e493-42d0-a549-811fb5fe7499-istio-envoy" pod "819ae12d-e493-42d0-a549-811fb5fe7499" (UID: "819ae12d-e493-42d0-a549-811fb5fe7499"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:46.838297 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.838259 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/819ae12d-e493-42d0-a549-811fb5fe7499-kube-api-access-6jh6k" pod "819ae12d-e493-42d0-a549-811fb5fe7499" (UID: "819ae12d-e493-42d0-a549-811fb5fe7499"). InnerVolumeSpecName "kube-api-access-6jh6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:49:46.924085 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.923994 4427 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/819ae12d-e493-42d0-a549-811fb5fe7499-istio-token\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:46.924085 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.924038 4427 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/819ae12d-e493-42d0-a549-811fb5fe7499-istio-envoy\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:46.924085 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.924053 4427 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/819ae12d-e493-42d0-a549-811fb5fe7499-istiod-ca-cert\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:46.924085 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.924070 4427 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/819ae12d-e493-42d0-a549-811fb5fe7499-credential-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:46.924085 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.924085 4427 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/819ae12d-e493-42d0-a549-811fb5fe7499-workload-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:46.924458 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.924099 4427 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/819ae12d-e493-42d0-a549-811fb5fe7499-istio-podinfo\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:46.924458 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.924113 4427 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/819ae12d-e493-42d0-a549-811fb5fe7499-workload-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:46.924458 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.924126 4427 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/819ae12d-e493-42d0-a549-811fb5fe7499-istio-data\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:46.924458 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:46.924140 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jh6k\" (UniqueName: \"kubernetes.io/projected/819ae12d-e493-42d0-a549-811fb5fe7499-kube-api-access-6jh6k\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:47.080550 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:47.080515 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-123986-86c99dc5d5-7n5p8" Nov 02 01:49:47.227617 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:47.227510 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f12da917-8a8f-445b-b939-0bab19110258-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f12da917-8a8f-445b-b939-0bab19110258-metrics-certs\") pod \"f12da917-8a8f-445b-b939-0bab19110258\" (UID: \"f12da917-8a8f-445b-b939-0bab19110258\") " Nov 02 01:49:47.227617 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:47.227567 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f12da917-8a8f-445b-b939-0bab19110258-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f12da917-8a8f-445b-b939-0bab19110258-maas-api-tls\") pod \"f12da917-8a8f-445b-b939-0bab19110258\" (UID: \"f12da917-8a8f-445b-b939-0bab19110258\") " Nov 02 01:49:47.227617 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:47.227614 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f12da917-8a8f-445b-b939-0bab19110258-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f12da917-8a8f-445b-b939-0bab19110258-openshift-service-ca\") pod \"f12da917-8a8f-445b-b939-0bab19110258\" (UID: \"f12da917-8a8f-445b-b939-0bab19110258\") " Nov 02 01:49:47.227923 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:47.227724 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f12da917-8a8f-445b-b939-0bab19110258-kube-api-access-kg9p6\" (UniqueName: \"kubernetes.io/projected/f12da917-8a8f-445b-b939-0bab19110258-kube-api-access-kg9p6\") pod \"f12da917-8a8f-445b-b939-0bab19110258\" (UID: \"f12da917-8a8f-445b-b939-0bab19110258\") " Nov 02 01:49:47.228476 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:47.228262 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f12da917-8a8f-445b-b939-0bab19110258-openshift-service-ca" pod "f12da917-8a8f-445b-b939-0bab19110258" (UID: "f12da917-8a8f-445b-b939-0bab19110258"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:49:47.230444 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:47.230413 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f12da917-8a8f-445b-b939-0bab19110258-metrics-certs" pod "f12da917-8a8f-445b-b939-0bab19110258" (UID: "f12da917-8a8f-445b-b939-0bab19110258"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:49:47.234891 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:47.234855 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f12da917-8a8f-445b-b939-0bab19110258-kube-api-access-kg9p6" pod "f12da917-8a8f-445b-b939-0bab19110258" (UID: "f12da917-8a8f-445b-b939-0bab19110258"). InnerVolumeSpecName "kube-api-access-kg9p6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:49:47.236776 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:47.236747 4427 generic.go:356] "Generic (PLEG): container finished" podID="819ae12d-e493-42d0-a549-811fb5fe7499" containerID="86bcb723f3ac09eb08da6a53144ab90378f17002034572135c475f435bf13c0c" exitCode=0 Nov 02 01:49:47.236986 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:47.236969 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv" Nov 02 01:49:47.237515 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:47.237488 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f12da917-8a8f-445b-b939-0bab19110258-maas-api-tls" pod "f12da917-8a8f-445b-b939-0bab19110258" (UID: "f12da917-8a8f-445b-b939-0bab19110258"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:49:47.237638 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:47.237544 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv" event={"ID":"819ae12d-e493-42d0-a549-811fb5fe7499","Type":"ContainerDied","Data":"86bcb723f3ac09eb08da6a53144ab90378f17002034572135c475f435bf13c0c"} Nov 02 01:49:47.237638 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:47.237598 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv" event={"ID":"819ae12d-e493-42d0-a549-811fb5fe7499","Type":"ContainerDied","Data":"fedfa286623e6c7075cecc115795fbdbd6d127ce9bd6417cd3d85d588d75aabb"} Nov 02 01:49:47.237638 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:47.237624 4427 scope.go:122] "RemoveContainer" containerID="86bcb723f3ac09eb08da6a53144ab90378f17002034572135c475f435bf13c0c" Nov 02 01:49:47.244480 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:47.243433 4427 generic.go:356] "Generic (PLEG): container finished" podID="f12da917-8a8f-445b-b939-0bab19110258" containerID="e7b24774ee2a5b38c99290040b1d604c87842dd18b5dc58f99e340169859ce05" exitCode=0 Nov 02 01:49:47.244480 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:47.243487 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-123986-86c99dc5d5-7n5p8" event={"ID":"f12da917-8a8f-445b-b939-0bab19110258","Type":"ContainerDied","Data":"e7b24774ee2a5b38c99290040b1d604c87842dd18b5dc58f99e340169859ce05"} Nov 02 01:49:47.244480 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:47.243530 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-123986-86c99dc5d5-7n5p8" event={"ID":"f12da917-8a8f-445b-b939-0bab19110258","Type":"ContainerDied","Data":"0404cbba2f9f05e1288a824169215049d1407db1cb46120797611e797ce28498"} Nov 02 01:49:47.244480 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:47.243605 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-123986-86c99dc5d5-7n5p8" Nov 02 01:49:47.253033 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:47.252939 4427 scope.go:122] "RemoveContainer" containerID="86bcb723f3ac09eb08da6a53144ab90378f17002034572135c475f435bf13c0c" Nov 02 01:49:47.253334 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:49:47.253254 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86bcb723f3ac09eb08da6a53144ab90378f17002034572135c475f435bf13c0c\": container with ID starting with 86bcb723f3ac09eb08da6a53144ab90378f17002034572135c475f435bf13c0c not found: ID does not exist" containerID="86bcb723f3ac09eb08da6a53144ab90378f17002034572135c475f435bf13c0c" Nov 02 01:49:47.253334 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:47.253295 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86bcb723f3ac09eb08da6a53144ab90378f17002034572135c475f435bf13c0c"} err="failed to get container status \"86bcb723f3ac09eb08da6a53144ab90378f17002034572135c475f435bf13c0c\": rpc error: code = NotFound desc = could not find container \"86bcb723f3ac09eb08da6a53144ab90378f17002034572135c475f435bf13c0c\": container with ID starting with 86bcb723f3ac09eb08da6a53144ab90378f17002034572135c475f435bf13c0c not found: ID does not exist" Nov 02 01:49:47.253334 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:47.253317 4427 scope.go:122] "RemoveContainer" containerID="e7b24774ee2a5b38c99290040b1d604c87842dd18b5dc58f99e340169859ce05" Nov 02 01:49:47.269951 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:47.269276 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv"] Nov 02 01:49:47.274727 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:47.274619 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-vxnqv"] Nov 02 01:49:47.274984 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:47.274965 4427 scope.go:122] "RemoveContainer" containerID="e7b24774ee2a5b38c99290040b1d604c87842dd18b5dc58f99e340169859ce05" Nov 02 01:49:47.275332 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:49:47.275309 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7b24774ee2a5b38c99290040b1d604c87842dd18b5dc58f99e340169859ce05\": container with ID starting with e7b24774ee2a5b38c99290040b1d604c87842dd18b5dc58f99e340169859ce05 not found: ID does not exist" containerID="e7b24774ee2a5b38c99290040b1d604c87842dd18b5dc58f99e340169859ce05" Nov 02 01:49:47.275465 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:47.275337 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7b24774ee2a5b38c99290040b1d604c87842dd18b5dc58f99e340169859ce05"} err="failed to get container status \"e7b24774ee2a5b38c99290040b1d604c87842dd18b5dc58f99e340169859ce05\": rpc error: code = NotFound desc = could not find container \"e7b24774ee2a5b38c99290040b1d604c87842dd18b5dc58f99e340169859ce05\": container with ID starting with e7b24774ee2a5b38c99290040b1d604c87842dd18b5dc58f99e340169859ce05 not found: ID does not exist" Nov 02 01:49:47.285838 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:47.285810 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-123986-86c99dc5d5-7n5p8"] Nov 02 01:49:47.291132 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:47.291106 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-123986-86c99dc5d5-7n5p8"] Nov 02 01:49:47.328911 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:47.328872 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg9p6\" (UniqueName: \"kubernetes.io/projected/f12da917-8a8f-445b-b939-0bab19110258-kube-api-access-kg9p6\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:47.328911 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:47.328911 4427 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f12da917-8a8f-445b-b939-0bab19110258-metrics-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:47.329089 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:47.328930 4427 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f12da917-8a8f-445b-b939-0bab19110258-maas-api-tls\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:47.329089 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:47.328945 4427 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f12da917-8a8f-445b-b939-0bab19110258-openshift-service-ca\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:47.533534 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:47.533447 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="819ae12d-e493-42d0-a549-811fb5fe7499" path="/var/lib/kubelet/pods/819ae12d-e493-42d0-a549-811fb5fe7499/volumes" Nov 02 01:49:47.533880 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:47.533865 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99cc3c61-f501-4a64-8f7e-0f41058aac68" path="/var/lib/kubelet/pods/99cc3c61-f501-4a64-8f7e-0f41058aac68/volumes" Nov 02 01:49:47.534228 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:47.534216 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f12da917-8a8f-445b-b939-0bab19110258" path="/var/lib/kubelet/pods/f12da917-8a8f-445b-b939-0bab19110258/volumes" Nov 02 01:49:48.959560 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:48.959535 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c" Nov 02 01:49:49.044191 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:49.044089 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c37ba466-a830-4246-832b-a8a4d0b5ab9f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c37ba466-a830-4246-832b-a8a4d0b5ab9f-istio-podinfo\") pod \"c37ba466-a830-4246-832b-a8a4d0b5ab9f\" (UID: \"c37ba466-a830-4246-832b-a8a4d0b5ab9f\") " Nov 02 01:49:49.044191 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:49.044136 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c37ba466-a830-4246-832b-a8a4d0b5ab9f-istio-token\" (UniqueName: \"kubernetes.io/projected/c37ba466-a830-4246-832b-a8a4d0b5ab9f-istio-token\") pod \"c37ba466-a830-4246-832b-a8a4d0b5ab9f\" (UID: \"c37ba466-a830-4246-832b-a8a4d0b5ab9f\") " Nov 02 01:49:49.044191 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:49.044178 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c37ba466-a830-4246-832b-a8a4d0b5ab9f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c37ba466-a830-4246-832b-a8a4d0b5ab9f-istio-envoy\") pod \"c37ba466-a830-4246-832b-a8a4d0b5ab9f\" (UID: \"c37ba466-a830-4246-832b-a8a4d0b5ab9f\") " Nov 02 01:49:49.044513 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:49.044216 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c37ba466-a830-4246-832b-a8a4d0b5ab9f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c37ba466-a830-4246-832b-a8a4d0b5ab9f-credential-socket\") pod \"c37ba466-a830-4246-832b-a8a4d0b5ab9f\" (UID: \"c37ba466-a830-4246-832b-a8a4d0b5ab9f\") " Nov 02 01:49:49.044513 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:49.044263 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c37ba466-a830-4246-832b-a8a4d0b5ab9f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c37ba466-a830-4246-832b-a8a4d0b5ab9f-workload-socket\") pod \"c37ba466-a830-4246-832b-a8a4d0b5ab9f\" (UID: \"c37ba466-a830-4246-832b-a8a4d0b5ab9f\") " Nov 02 01:49:49.044513 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:49.044307 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c37ba466-a830-4246-832b-a8a4d0b5ab9f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c37ba466-a830-4246-832b-a8a4d0b5ab9f-workload-certs\") pod \"c37ba466-a830-4246-832b-a8a4d0b5ab9f\" (UID: \"c37ba466-a830-4246-832b-a8a4d0b5ab9f\") " Nov 02 01:49:49.044513 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:49.044329 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c37ba466-a830-4246-832b-a8a4d0b5ab9f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c37ba466-a830-4246-832b-a8a4d0b5ab9f-istio-data\") pod \"c37ba466-a830-4246-832b-a8a4d0b5ab9f\" (UID: \"c37ba466-a830-4246-832b-a8a4d0b5ab9f\") " Nov 02 01:49:49.044513 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:49.044367 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c37ba466-a830-4246-832b-a8a4d0b5ab9f-kube-api-access-gx9f9\" (UniqueName: \"kubernetes.io/projected/c37ba466-a830-4246-832b-a8a4d0b5ab9f-kube-api-access-gx9f9\") pod \"c37ba466-a830-4246-832b-a8a4d0b5ab9f\" (UID: \"c37ba466-a830-4246-832b-a8a4d0b5ab9f\") " Nov 02 01:49:49.044513 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:49.044416 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c37ba466-a830-4246-832b-a8a4d0b5ab9f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c37ba466-a830-4246-832b-a8a4d0b5ab9f-istiod-ca-cert\") pod \"c37ba466-a830-4246-832b-a8a4d0b5ab9f\" (UID: \"c37ba466-a830-4246-832b-a8a4d0b5ab9f\") " Nov 02 01:49:49.044811 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:49.044688 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c37ba466-a830-4246-832b-a8a4d0b5ab9f-workload-certs" pod "c37ba466-a830-4246-832b-a8a4d0b5ab9f" (UID: "c37ba466-a830-4246-832b-a8a4d0b5ab9f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:49.044811 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:49.044697 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c37ba466-a830-4246-832b-a8a4d0b5ab9f-istio-data" pod "c37ba466-a830-4246-832b-a8a4d0b5ab9f" (UID: "c37ba466-a830-4246-832b-a8a4d0b5ab9f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:49.044932 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:49.044814 4427 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c37ba466-a830-4246-832b-a8a4d0b5ab9f-workload-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:49.044932 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:49.044835 4427 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c37ba466-a830-4246-832b-a8a4d0b5ab9f-istio-data\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:49.045109 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:49.045071 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c37ba466-a830-4246-832b-a8a4d0b5ab9f-workload-socket" pod "c37ba466-a830-4246-832b-a8a4d0b5ab9f" (UID: "c37ba466-a830-4246-832b-a8a4d0b5ab9f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:49.045109 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:49.045096 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c37ba466-a830-4246-832b-a8a4d0b5ab9f-istiod-ca-cert" pod "c37ba466-a830-4246-832b-a8a4d0b5ab9f" (UID: "c37ba466-a830-4246-832b-a8a4d0b5ab9f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:49:49.045229 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:49.045182 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c37ba466-a830-4246-832b-a8a4d0b5ab9f-credential-socket" pod "c37ba466-a830-4246-832b-a8a4d0b5ab9f" (UID: "c37ba466-a830-4246-832b-a8a4d0b5ab9f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:49.047415 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:49.047354 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c37ba466-a830-4246-832b-a8a4d0b5ab9f-istio-envoy" pod "c37ba466-a830-4246-832b-a8a4d0b5ab9f" (UID: "c37ba466-a830-4246-832b-a8a4d0b5ab9f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:49.047415 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:49.047351 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c37ba466-a830-4246-832b-a8a4d0b5ab9f-kube-api-access-gx9f9" pod "c37ba466-a830-4246-832b-a8a4d0b5ab9f" (UID: "c37ba466-a830-4246-832b-a8a4d0b5ab9f"). InnerVolumeSpecName "kube-api-access-gx9f9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:49:49.049178 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:49.048519 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c37ba466-a830-4246-832b-a8a4d0b5ab9f-istio-podinfo" pod "c37ba466-a830-4246-832b-a8a4d0b5ab9f" (UID: "c37ba466-a830-4246-832b-a8a4d0b5ab9f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:49:49.049178 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:49.048538 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c37ba466-a830-4246-832b-a8a4d0b5ab9f-istio-token" pod "c37ba466-a830-4246-832b-a8a4d0b5ab9f" (UID: "c37ba466-a830-4246-832b-a8a4d0b5ab9f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:49:49.145426 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:49.145353 4427 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c37ba466-a830-4246-832b-a8a4d0b5ab9f-istio-podinfo\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:49.145426 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:49.145414 4427 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c37ba466-a830-4246-832b-a8a4d0b5ab9f-istio-token\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:49.145426 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:49.145424 4427 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c37ba466-a830-4246-832b-a8a4d0b5ab9f-istio-envoy\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:49.145426 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:49.145433 4427 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c37ba466-a830-4246-832b-a8a4d0b5ab9f-credential-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:49.145711 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:49.145442 4427 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c37ba466-a830-4246-832b-a8a4d0b5ab9f-workload-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:49.145711 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:49.145451 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx9f9\" (UniqueName: \"kubernetes.io/projected/c37ba466-a830-4246-832b-a8a4d0b5ab9f-kube-api-access-gx9f9\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:49.145711 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:49.145461 4427 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c37ba466-a830-4246-832b-a8a4d0b5ab9f-istiod-ca-cert\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:49.256720 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:49.256681 4427 generic.go:356] "Generic (PLEG): container finished" podID="c37ba466-a830-4246-832b-a8a4d0b5ab9f" containerID="73d46afa0f9cf0460c0eb36b5ffc4af31c63614272d021f283b01568394d56f8" exitCode=0 Nov 02 01:49:49.256904 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:49.256730 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c" Nov 02 01:49:49.256904 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:49.256740 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c" event={"ID":"c37ba466-a830-4246-832b-a8a4d0b5ab9f","Type":"ContainerDied","Data":"73d46afa0f9cf0460c0eb36b5ffc4af31c63614272d021f283b01568394d56f8"} Nov 02 01:49:49.256904 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:49.256795 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c" event={"ID":"c37ba466-a830-4246-832b-a8a4d0b5ab9f","Type":"ContainerDied","Data":"8adb3a232408bf710b9bcef8bf770878b59e42ee9287210113b732ddd99ae1c2"} Nov 02 01:49:49.256904 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:49.256818 4427 scope.go:122] "RemoveContainer" containerID="73d46afa0f9cf0460c0eb36b5ffc4af31c63614272d021f283b01568394d56f8" Nov 02 01:49:49.266627 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:49.266604 4427 scope.go:122] "RemoveContainer" containerID="73d46afa0f9cf0460c0eb36b5ffc4af31c63614272d021f283b01568394d56f8" Nov 02 01:49:49.266900 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:49:49.266881 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73d46afa0f9cf0460c0eb36b5ffc4af31c63614272d021f283b01568394d56f8\": container with ID starting with 73d46afa0f9cf0460c0eb36b5ffc4af31c63614272d021f283b01568394d56f8 not found: ID does not exist" containerID="73d46afa0f9cf0460c0eb36b5ffc4af31c63614272d021f283b01568394d56f8" Nov 02 01:49:49.266945 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:49.266911 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73d46afa0f9cf0460c0eb36b5ffc4af31c63614272d021f283b01568394d56f8"} err="failed to get container status \"73d46afa0f9cf0460c0eb36b5ffc4af31c63614272d021f283b01568394d56f8\": rpc error: code = NotFound desc = could not find container \"73d46afa0f9cf0460c0eb36b5ffc4af31c63614272d021f283b01568394d56f8\": container with ID starting with 73d46afa0f9cf0460c0eb36b5ffc4af31c63614272d021f283b01568394d56f8 not found: ID does not exist" Nov 02 01:49:49.279764 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:49.279740 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c"] Nov 02 01:49:49.283897 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:49.283873 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ca1c53-openshift-default-5b5859dc89-xfb6c"] Nov 02 01:49:49.532222 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:49.532187 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c37ba466-a830-4246-832b-a8a4d0b5ab9f" path="/var/lib/kubelet/pods/c37ba466-a830-4246-832b-a8a4d0b5ab9f/volumes" Nov 02 01:49:53.867117 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:53.867076 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf"] Nov 02 01:49:53.887148 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:53.887104 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf" podUID="b880c081-4957-4151-9e1f-fb8f397dedb3" containerName="istio-proxy" containerID="cri-o://16e18f9cf7042ff833baa19cc64cabba064002c9fb876efdd1927e2c6e07a2a8" gracePeriod=30 Nov 02 01:49:56.503434 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:56.503391 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-587977945b-ktcgw"] Nov 02 01:49:56.503948 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:56.503713 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-587977945b-ktcgw" podUID="bcf15fa4-15f1-4dc9-8755-9bcccfdc846b" containerName="manager" containerID="cri-o://f7328556dd4f638e90722886ed40b7848bcaae2ee83c6f45a36405d8ee6166f2" gracePeriod=30 Nov 02 01:49:56.760114 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:56.760051 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-587977945b-ktcgw" Nov 02 01:49:56.811768 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:56.811728 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcf15fa4-15f1-4dc9-8755-9bcccfdc846b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcf15fa4-15f1-4dc9-8755-9bcccfdc846b-metrics-certs\") pod \"bcf15fa4-15f1-4dc9-8755-9bcccfdc846b\" (UID: \"bcf15fa4-15f1-4dc9-8755-9bcccfdc846b\") " Nov 02 01:49:56.811768 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:56.811780 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcf15fa4-15f1-4dc9-8755-9bcccfdc846b-webhook-cert\" (UniqueName: \"kubernetes.io/secret/bcf15fa4-15f1-4dc9-8755-9bcccfdc846b-webhook-cert\") pod \"bcf15fa4-15f1-4dc9-8755-9bcccfdc846b\" (UID: \"bcf15fa4-15f1-4dc9-8755-9bcccfdc846b\") " Nov 02 01:49:56.811992 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:56.811818 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcf15fa4-15f1-4dc9-8755-9bcccfdc846b-kube-api-access-dh8vb\" (UniqueName: \"kubernetes.io/projected/bcf15fa4-15f1-4dc9-8755-9bcccfdc846b-kube-api-access-dh8vb\") pod \"bcf15fa4-15f1-4dc9-8755-9bcccfdc846b\" (UID: \"bcf15fa4-15f1-4dc9-8755-9bcccfdc846b\") " Nov 02 01:49:56.814361 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:56.814325 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcf15fa4-15f1-4dc9-8755-9bcccfdc846b-kube-api-access-dh8vb" pod "bcf15fa4-15f1-4dc9-8755-9bcccfdc846b" (UID: "bcf15fa4-15f1-4dc9-8755-9bcccfdc846b"). InnerVolumeSpecName "kube-api-access-dh8vb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:49:56.814630 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:56.814609 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcf15fa4-15f1-4dc9-8755-9bcccfdc846b-webhook-cert" pod "bcf15fa4-15f1-4dc9-8755-9bcccfdc846b" (UID: "bcf15fa4-15f1-4dc9-8755-9bcccfdc846b"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:49:56.814715 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:56.814688 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcf15fa4-15f1-4dc9-8755-9bcccfdc846b-metrics-certs" pod "bcf15fa4-15f1-4dc9-8755-9bcccfdc846b" (UID: "bcf15fa4-15f1-4dc9-8755-9bcccfdc846b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:49:56.913501 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:56.913461 4427 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcf15fa4-15f1-4dc9-8755-9bcccfdc846b-metrics-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:56.913501 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:56.913492 4427 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bcf15fa4-15f1-4dc9-8755-9bcccfdc846b-webhook-cert\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:56.913501 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:56.913502 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dh8vb\" (UniqueName: \"kubernetes.io/projected/bcf15fa4-15f1-4dc9-8755-9bcccfdc846b-kube-api-access-dh8vb\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:57.308996 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:57.308956 4427 generic.go:356] "Generic (PLEG): container finished" podID="bcf15fa4-15f1-4dc9-8755-9bcccfdc846b" containerID="f7328556dd4f638e90722886ed40b7848bcaae2ee83c6f45a36405d8ee6166f2" exitCode=0 Nov 02 01:49:57.309216 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:57.309013 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-587977945b-ktcgw" Nov 02 01:49:57.309216 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:57.309013 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-587977945b-ktcgw" event={"ID":"bcf15fa4-15f1-4dc9-8755-9bcccfdc846b","Type":"ContainerDied","Data":"f7328556dd4f638e90722886ed40b7848bcaae2ee83c6f45a36405d8ee6166f2"} Nov 02 01:49:57.309216 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:57.309089 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-587977945b-ktcgw" event={"ID":"bcf15fa4-15f1-4dc9-8755-9bcccfdc846b","Type":"ContainerDied","Data":"029c0899ce6e99f9caea3937c30a40a1c947deb51dca3cae3c024ac90533ff7d"} Nov 02 01:49:57.309216 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:57.309121 4427 scope.go:122] "RemoveContainer" containerID="f7328556dd4f638e90722886ed40b7848bcaae2ee83c6f45a36405d8ee6166f2" Nov 02 01:49:57.320220 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:57.320201 4427 scope.go:122] "RemoveContainer" containerID="c22144c1728a9529c65802199385bc79bf0d1822b08839703133f102f0153502" Nov 02 01:49:57.333064 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:57.333037 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-587977945b-ktcgw"] Nov 02 01:49:57.337414 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:57.337292 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-587977945b-ktcgw"] Nov 02 01:49:57.337500 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:57.337484 4427 scope.go:122] "RemoveContainer" containerID="f7328556dd4f638e90722886ed40b7848bcaae2ee83c6f45a36405d8ee6166f2" Nov 02 01:49:57.337761 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:49:57.337745 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7328556dd4f638e90722886ed40b7848bcaae2ee83c6f45a36405d8ee6166f2\": container with ID starting with f7328556dd4f638e90722886ed40b7848bcaae2ee83c6f45a36405d8ee6166f2 not found: ID does not exist" containerID="f7328556dd4f638e90722886ed40b7848bcaae2ee83c6f45a36405d8ee6166f2" Nov 02 01:49:57.337807 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:57.337769 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7328556dd4f638e90722886ed40b7848bcaae2ee83c6f45a36405d8ee6166f2"} err="failed to get container status \"f7328556dd4f638e90722886ed40b7848bcaae2ee83c6f45a36405d8ee6166f2\": rpc error: code = NotFound desc = could not find container \"f7328556dd4f638e90722886ed40b7848bcaae2ee83c6f45a36405d8ee6166f2\": container with ID starting with f7328556dd4f638e90722886ed40b7848bcaae2ee83c6f45a36405d8ee6166f2 not found: ID does not exist" Nov 02 01:49:57.337807 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:57.337785 4427 scope.go:122] "RemoveContainer" containerID="c22144c1728a9529c65802199385bc79bf0d1822b08839703133f102f0153502" Nov 02 01:49:57.338044 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:49:57.338026 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c22144c1728a9529c65802199385bc79bf0d1822b08839703133f102f0153502\": container with ID starting with c22144c1728a9529c65802199385bc79bf0d1822b08839703133f102f0153502 not found: ID does not exist" containerID="c22144c1728a9529c65802199385bc79bf0d1822b08839703133f102f0153502" Nov 02 01:49:57.338101 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:57.338047 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c22144c1728a9529c65802199385bc79bf0d1822b08839703133f102f0153502"} err="failed to get container status \"c22144c1728a9529c65802199385bc79bf0d1822b08839703133f102f0153502\": rpc error: code = NotFound desc = could not find container \"c22144c1728a9529c65802199385bc79bf0d1822b08839703133f102f0153502\": container with ID starting with c22144c1728a9529c65802199385bc79bf0d1822b08839703133f102f0153502 not found: ID does not exist" Nov 02 01:49:57.535043 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:57.535014 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcf15fa4-15f1-4dc9-8755-9bcccfdc846b" path="/var/lib/kubelet/pods/bcf15fa4-15f1-4dc9-8755-9bcccfdc846b/volumes" Nov 02 01:49:59.197849 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:59.197815 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf" Nov 02 01:49:59.236902 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:59.236865 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b880c081-4957-4151-9e1f-fb8f397dedb3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b880c081-4957-4151-9e1f-fb8f397dedb3-credential-socket\") pod \"b880c081-4957-4151-9e1f-fb8f397dedb3\" (UID: \"b880c081-4957-4151-9e1f-fb8f397dedb3\") " Nov 02 01:49:59.236902 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:59.236908 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b880c081-4957-4151-9e1f-fb8f397dedb3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b880c081-4957-4151-9e1f-fb8f397dedb3-istio-envoy\") pod \"b880c081-4957-4151-9e1f-fb8f397dedb3\" (UID: \"b880c081-4957-4151-9e1f-fb8f397dedb3\") " Nov 02 01:49:59.237207 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:59.236932 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b880c081-4957-4151-9e1f-fb8f397dedb3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b880c081-4957-4151-9e1f-fb8f397dedb3-workload-socket\") pod \"b880c081-4957-4151-9e1f-fb8f397dedb3\" (UID: \"b880c081-4957-4151-9e1f-fb8f397dedb3\") " Nov 02 01:49:59.237207 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:59.236985 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b880c081-4957-4151-9e1f-fb8f397dedb3-istio-token\" (UniqueName: \"kubernetes.io/projected/b880c081-4957-4151-9e1f-fb8f397dedb3-istio-token\") pod \"b880c081-4957-4151-9e1f-fb8f397dedb3\" (UID: \"b880c081-4957-4151-9e1f-fb8f397dedb3\") " Nov 02 01:49:59.237207 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:59.237053 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b880c081-4957-4151-9e1f-fb8f397dedb3-kube-api-access-l5nq9\" (UniqueName: \"kubernetes.io/projected/b880c081-4957-4151-9e1f-fb8f397dedb3-kube-api-access-l5nq9\") pod \"b880c081-4957-4151-9e1f-fb8f397dedb3\" (UID: \"b880c081-4957-4151-9e1f-fb8f397dedb3\") " Nov 02 01:49:59.237207 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:59.237072 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b880c081-4957-4151-9e1f-fb8f397dedb3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b880c081-4957-4151-9e1f-fb8f397dedb3-istio-podinfo\") pod \"b880c081-4957-4151-9e1f-fb8f397dedb3\" (UID: \"b880c081-4957-4151-9e1f-fb8f397dedb3\") " Nov 02 01:49:59.237207 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:59.237125 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b880c081-4957-4151-9e1f-fb8f397dedb3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b880c081-4957-4151-9e1f-fb8f397dedb3-istiod-ca-cert\") pod \"b880c081-4957-4151-9e1f-fb8f397dedb3\" (UID: \"b880c081-4957-4151-9e1f-fb8f397dedb3\") " Nov 02 01:49:59.237207 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:59.237158 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b880c081-4957-4151-9e1f-fb8f397dedb3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b880c081-4957-4151-9e1f-fb8f397dedb3-workload-certs\") pod \"b880c081-4957-4151-9e1f-fb8f397dedb3\" (UID: \"b880c081-4957-4151-9e1f-fb8f397dedb3\") " Nov 02 01:49:59.237207 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:59.237204 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b880c081-4957-4151-9e1f-fb8f397dedb3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b880c081-4957-4151-9e1f-fb8f397dedb3-istio-data\") pod \"b880c081-4957-4151-9e1f-fb8f397dedb3\" (UID: \"b880c081-4957-4151-9e1f-fb8f397dedb3\") " Nov 02 01:49:59.237668 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:59.237638 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b880c081-4957-4151-9e1f-fb8f397dedb3-istio-data" pod "b880c081-4957-4151-9e1f-fb8f397dedb3" (UID: "b880c081-4957-4151-9e1f-fb8f397dedb3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:59.237828 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:59.237803 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b880c081-4957-4151-9e1f-fb8f397dedb3-workload-socket" pod "b880c081-4957-4151-9e1f-fb8f397dedb3" (UID: "b880c081-4957-4151-9e1f-fb8f397dedb3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:59.238003 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:59.237954 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b880c081-4957-4151-9e1f-fb8f397dedb3-credential-socket" pod "b880c081-4957-4151-9e1f-fb8f397dedb3" (UID: "b880c081-4957-4151-9e1f-fb8f397dedb3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:59.238265 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:59.238232 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b880c081-4957-4151-9e1f-fb8f397dedb3-workload-certs" pod "b880c081-4957-4151-9e1f-fb8f397dedb3" (UID: "b880c081-4957-4151-9e1f-fb8f397dedb3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:59.238404 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:59.238238 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b880c081-4957-4151-9e1f-fb8f397dedb3-istiod-ca-cert" pod "b880c081-4957-4151-9e1f-fb8f397dedb3" (UID: "b880c081-4957-4151-9e1f-fb8f397dedb3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:49:59.240452 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:59.240427 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b880c081-4957-4151-9e1f-fb8f397dedb3-kube-api-access-l5nq9" pod "b880c081-4957-4151-9e1f-fb8f397dedb3" (UID: "b880c081-4957-4151-9e1f-fb8f397dedb3"). InnerVolumeSpecName "kube-api-access-l5nq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:49:59.240842 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:59.240507 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b880c081-4957-4151-9e1f-fb8f397dedb3-istio-envoy" pod "b880c081-4957-4151-9e1f-fb8f397dedb3" (UID: "b880c081-4957-4151-9e1f-fb8f397dedb3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:49:59.242057 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:59.242035 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b880c081-4957-4151-9e1f-fb8f397dedb3-istio-token" pod "b880c081-4957-4151-9e1f-fb8f397dedb3" (UID: "b880c081-4957-4151-9e1f-fb8f397dedb3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:49:59.242181 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:59.242162 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b880c081-4957-4151-9e1f-fb8f397dedb3-istio-podinfo" pod "b880c081-4957-4151-9e1f-fb8f397dedb3" (UID: "b880c081-4957-4151-9e1f-fb8f397dedb3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:49:59.327000 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:59.326961 4427 generic.go:356] "Generic (PLEG): container finished" podID="b880c081-4957-4151-9e1f-fb8f397dedb3" containerID="16e18f9cf7042ff833baa19cc64cabba064002c9fb876efdd1927e2c6e07a2a8" exitCode=0 Nov 02 01:49:59.327162 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:59.327048 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf" event={"ID":"b880c081-4957-4151-9e1f-fb8f397dedb3","Type":"ContainerDied","Data":"16e18f9cf7042ff833baa19cc64cabba064002c9fb876efdd1927e2c6e07a2a8"} Nov 02 01:49:59.327162 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:59.327119 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf" event={"ID":"b880c081-4957-4151-9e1f-fb8f397dedb3","Type":"ContainerDied","Data":"bb8535ae02beb1f941e50178793aa4418bef0b5be7435322579b67172a1cb123"} Nov 02 01:49:59.327162 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:59.327061 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf" Nov 02 01:49:59.327162 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:59.327156 4427 scope.go:122] "RemoveContainer" containerID="16e18f9cf7042ff833baa19cc64cabba064002c9fb876efdd1927e2c6e07a2a8" Nov 02 01:49:59.339991 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:59.339960 4427 scope.go:122] "RemoveContainer" containerID="b5107dd9a7488440da3a86139d85e7887cda8bd9a4f18ed1e23170d1d29f81d0" Nov 02 01:49:59.340471 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:59.340451 4427 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b880c081-4957-4151-9e1f-fb8f397dedb3-istiod-ca-cert\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:59.340471 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:59.340471 4427 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b880c081-4957-4151-9e1f-fb8f397dedb3-workload-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:59.340649 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:59.340483 4427 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b880c081-4957-4151-9e1f-fb8f397dedb3-istio-data\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:59.340649 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:59.340498 4427 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b880c081-4957-4151-9e1f-fb8f397dedb3-credential-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:59.340649 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:59.340507 4427 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b880c081-4957-4151-9e1f-fb8f397dedb3-istio-envoy\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:59.340649 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:59.340517 4427 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b880c081-4957-4151-9e1f-fb8f397dedb3-workload-socket\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:59.340649 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:59.340531 4427 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b880c081-4957-4151-9e1f-fb8f397dedb3-istio-token\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:59.340649 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:59.340543 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5nq9\" (UniqueName: \"kubernetes.io/projected/b880c081-4957-4151-9e1f-fb8f397dedb3-kube-api-access-l5nq9\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:59.340649 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:59.340558 4427 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b880c081-4957-4151-9e1f-fb8f397dedb3-istio-podinfo\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:49:59.349172 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:59.349107 4427 scope.go:122] "RemoveContainer" containerID="16e18f9cf7042ff833baa19cc64cabba064002c9fb876efdd1927e2c6e07a2a8" Nov 02 01:49:59.350123 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:49:59.350101 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16e18f9cf7042ff833baa19cc64cabba064002c9fb876efdd1927e2c6e07a2a8\": container with ID starting with 16e18f9cf7042ff833baa19cc64cabba064002c9fb876efdd1927e2c6e07a2a8 not found: ID does not exist" containerID="16e18f9cf7042ff833baa19cc64cabba064002c9fb876efdd1927e2c6e07a2a8" Nov 02 01:49:59.350210 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:59.350131 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16e18f9cf7042ff833baa19cc64cabba064002c9fb876efdd1927e2c6e07a2a8"} err="failed to get container status \"16e18f9cf7042ff833baa19cc64cabba064002c9fb876efdd1927e2c6e07a2a8\": rpc error: code = NotFound desc = could not find container \"16e18f9cf7042ff833baa19cc64cabba064002c9fb876efdd1927e2c6e07a2a8\": container with ID starting with 16e18f9cf7042ff833baa19cc64cabba064002c9fb876efdd1927e2c6e07a2a8 not found: ID does not exist" Nov 02 01:49:59.350210 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:59.350147 4427 scope.go:122] "RemoveContainer" containerID="b5107dd9a7488440da3a86139d85e7887cda8bd9a4f18ed1e23170d1d29f81d0" Nov 02 01:49:59.350460 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:49:59.350440 4427 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5107dd9a7488440da3a86139d85e7887cda8bd9a4f18ed1e23170d1d29f81d0\": container with ID starting with b5107dd9a7488440da3a86139d85e7887cda8bd9a4f18ed1e23170d1d29f81d0 not found: ID does not exist" containerID="b5107dd9a7488440da3a86139d85e7887cda8bd9a4f18ed1e23170d1d29f81d0" Nov 02 01:49:59.350533 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:59.350465 4427 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5107dd9a7488440da3a86139d85e7887cda8bd9a4f18ed1e23170d1d29f81d0"} err="failed to get container status \"b5107dd9a7488440da3a86139d85e7887cda8bd9a4f18ed1e23170d1d29f81d0\": rpc error: code = NotFound desc = could not find container \"b5107dd9a7488440da3a86139d85e7887cda8bd9a4f18ed1e23170d1d29f81d0\": container with ID starting with b5107dd9a7488440da3a86139d85e7887cda8bd9a4f18ed1e23170d1d29f81d0 not found: ID does not exist" Nov 02 01:49:59.351633 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:59.351614 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf"] Nov 02 01:49:59.355064 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:59.355044 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-123986-openshift-default-5d89d69677-tdcdf"] Nov 02 01:49:59.533868 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:49:59.533777 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b880c081-4957-4151-9e1f-fb8f397dedb3" path="/var/lib/kubelet/pods/b880c081-4957-4151-9e1f-fb8f397dedb3/volumes" Nov 02 01:50:01.581890 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:50:01.581850 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:50:25.761988 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.761904 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kgg42"] Nov 02 01:50:25.762559 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.762519 4427 cpu_manager.go:414] "containerMap: removing container" podUID="bcf15fa4-15f1-4dc9-8755-9bcccfdc846b" containerName="manager" Nov 02 01:50:25.762559 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.762540 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcf15fa4-15f1-4dc9-8755-9bcccfdc846b" containerName="manager" Nov 02 01:50:25.762654 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.762561 4427 cpu_manager.go:414] "containerMap: removing container" podUID="fe3cde15-cd40-4b6a-9144-f96a05179a27" containerName="maas-api" Nov 02 01:50:25.762654 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.762568 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe3cde15-cd40-4b6a-9144-f96a05179a27" containerName="maas-api" Nov 02 01:50:25.762654 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.762579 4427 cpu_manager.go:414] "containerMap: removing container" podUID="b880c081-4957-4151-9e1f-fb8f397dedb3" containerName="istio-proxy" Nov 02 01:50:25.762654 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.762587 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b880c081-4957-4151-9e1f-fb8f397dedb3" containerName="istio-proxy" Nov 02 01:50:25.762654 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.762597 4427 cpu_manager.go:414] "containerMap: removing container" podUID="819ae12d-e493-42d0-a549-811fb5fe7499" containerName="istio-proxy" Nov 02 01:50:25.762654 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.762605 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="819ae12d-e493-42d0-a549-811fb5fe7499" containerName="istio-proxy" Nov 02 01:50:25.762654 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.762626 4427 cpu_manager.go:414] "containerMap: removing container" podUID="f67c954a-ce5b-41d9-8d99-7d126bd89d59" containerName="main" Nov 02 01:50:25.762654 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.762636 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f67c954a-ce5b-41d9-8d99-7d126bd89d59" containerName="main" Nov 02 01:50:25.762654 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.762648 4427 cpu_manager.go:414] "containerMap: removing container" podUID="c37ba466-a830-4246-832b-a8a4d0b5ab9f" containerName="istio-proxy" Nov 02 01:50:25.762654 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.762655 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c37ba466-a830-4246-832b-a8a4d0b5ab9f" containerName="istio-proxy" Nov 02 01:50:25.762995 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.762667 4427 cpu_manager.go:414] "containerMap: removing container" podUID="fca29702-c34c-4933-a93a-143c5481ba3a" containerName="payload-pre-processing" Nov 02 01:50:25.762995 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.762677 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fca29702-c34c-4933-a93a-143c5481ba3a" containerName="payload-pre-processing" Nov 02 01:50:25.762995 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.762690 4427 cpu_manager.go:414] "containerMap: removing container" podUID="d5cb7660-e2aa-48a2-9eac-508cd83f93cb" containerName="istio-proxy" Nov 02 01:50:25.762995 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.762697 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5cb7660-e2aa-48a2-9eac-508cd83f93cb" containerName="istio-proxy" Nov 02 01:50:25.762995 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.762706 4427 cpu_manager.go:414] "containerMap: removing container" podUID="317a9ada-6f39-4dee-8945-9cc2dd1c18eb" containerName="payload-pre-processing" Nov 02 01:50:25.762995 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.762714 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="317a9ada-6f39-4dee-8945-9cc2dd1c18eb" containerName="payload-pre-processing" Nov 02 01:50:25.762995 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.762726 4427 cpu_manager.go:414] "containerMap: removing container" podUID="b880c081-4957-4151-9e1f-fb8f397dedb3" containerName="istio-proxy" Nov 02 01:50:25.762995 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.762733 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b880c081-4957-4151-9e1f-fb8f397dedb3" containerName="istio-proxy" Nov 02 01:50:25.762995 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.762743 4427 cpu_manager.go:414] "containerMap: removing container" podUID="d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb" containerName="main" Nov 02 01:50:25.762995 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.762749 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb" containerName="main" Nov 02 01:50:25.762995 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.762759 4427 cpu_manager.go:414] "containerMap: removing container" podUID="1d6df883-72f0-4727-acbc-72bb9bd02f05" containerName="istio-proxy" Nov 02 01:50:25.762995 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.762766 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d6df883-72f0-4727-acbc-72bb9bd02f05" containerName="istio-proxy" Nov 02 01:50:25.762995 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.762778 4427 cpu_manager.go:414] "containerMap: removing container" podUID="8f9b0669-291d-43fa-9511-d74948b03684" containerName="istio-proxy" Nov 02 01:50:25.762995 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.762785 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f9b0669-291d-43fa-9511-d74948b03684" containerName="istio-proxy" Nov 02 01:50:25.762995 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.762801 4427 cpu_manager.go:414] "containerMap: removing container" podUID="392681ce-fe14-4518-9697-efb94f82efdc" containerName="istio-proxy" Nov 02 01:50:25.762995 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.762809 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="392681ce-fe14-4518-9697-efb94f82efdc" containerName="istio-proxy" Nov 02 01:50:25.762995 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.762820 4427 cpu_manager.go:414] "containerMap: removing container" podUID="d133c468-9c0c-4d15-8b0e-9292b27944f6" containerName="payload-processing" Nov 02 01:50:25.762995 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.762827 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d133c468-9c0c-4d15-8b0e-9292b27944f6" containerName="payload-processing" Nov 02 01:50:25.762995 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.762840 4427 cpu_manager.go:414] "containerMap: removing container" podUID="8f9b0669-291d-43fa-9511-d74948b03684" containerName="istio-proxy" Nov 02 01:50:25.762995 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.762847 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f9b0669-291d-43fa-9511-d74948b03684" containerName="istio-proxy" Nov 02 01:50:25.762995 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.762860 4427 cpu_manager.go:414] "containerMap: removing container" podUID="eb3258a9-adfb-46de-a0ab-7b4649948689" containerName="istio-proxy" Nov 02 01:50:25.762995 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.762867 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb3258a9-adfb-46de-a0ab-7b4649948689" containerName="istio-proxy" Nov 02 01:50:25.762995 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.762878 4427 cpu_manager.go:414] "containerMap: removing container" podUID="17e05811-e6f0-4c5b-be43-d39429d1af11" containerName="payload-pre-processing" Nov 02 01:50:25.762995 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.762885 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17e05811-e6f0-4c5b-be43-d39429d1af11" containerName="payload-pre-processing" Nov 02 01:50:25.762995 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.762897 4427 cpu_manager.go:414] "containerMap: removing container" podUID="99cc3c61-f501-4a64-8f7e-0f41058aac68" containerName="istio-proxy" Nov 02 01:50:25.762995 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.762904 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99cc3c61-f501-4a64-8f7e-0f41058aac68" containerName="istio-proxy" Nov 02 01:50:25.762995 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.762914 4427 cpu_manager.go:414] "containerMap: removing container" podUID="aceec71b-ffcb-47e7-964e-84bc8405f2de" containerName="payload-pre-processing" Nov 02 01:50:25.762995 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.762921 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aceec71b-ffcb-47e7-964e-84bc8405f2de" containerName="payload-pre-processing" Nov 02 01:50:25.762995 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.762938 4427 cpu_manager.go:414] "containerMap: removing container" podUID="9c056798-2cf8-47b1-9829-67cebda7a248" containerName="main" Nov 02 01:50:25.762995 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.762945 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c056798-2cf8-47b1-9829-67cebda7a248" containerName="main" Nov 02 01:50:25.762995 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.762965 4427 cpu_manager.go:414] "containerMap: removing container" podUID="79fb94dd-4426-43ed-af58-cfa762e77b13" containerName="payload-processing" Nov 02 01:50:25.762995 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.762972 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79fb94dd-4426-43ed-af58-cfa762e77b13" containerName="payload-processing" Nov 02 01:50:25.762995 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.762990 4427 cpu_manager.go:414] "containerMap: removing container" podUID="f12da917-8a8f-445b-b939-0bab19110258" containerName="maas-api" Nov 02 01:50:25.762995 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.762997 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f12da917-8a8f-445b-b939-0bab19110258" containerName="maas-api" Nov 02 01:50:25.762995 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.763006 4427 cpu_manager.go:414] "containerMap: removing container" podUID="7085c0ab-7ee3-4665-abad-2977c25f68ba" containerName="main" Nov 02 01:50:25.762995 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.763013 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7085c0ab-7ee3-4665-abad-2977c25f68ba" containerName="main" Nov 02 01:50:25.764148 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.763023 4427 cpu_manager.go:414] "containerMap: removing container" podUID="af28d18e-199e-4c22-bb6d-9b83e7e0473a" containerName="revoke-keys" Nov 02 01:50:25.764148 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.763030 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af28d18e-199e-4c22-bb6d-9b83e7e0473a" containerName="revoke-keys" Nov 02 01:50:25.764148 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.763042 4427 cpu_manager.go:414] "containerMap: removing container" podUID="f31dc5a1-ea54-4858-ac4c-2b990192ee4c" containerName="istio-proxy" Nov 02 01:50:25.764148 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.763049 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f31dc5a1-ea54-4858-ac4c-2b990192ee4c" containerName="istio-proxy" Nov 02 01:50:25.764148 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.763173 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcf15fa4-15f1-4dc9-8755-9bcccfdc846b" containerName="manager" Nov 02 01:50:25.764148 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.763186 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe3cde15-cd40-4b6a-9144-f96a05179a27" containerName="maas-api" Nov 02 01:50:25.764148 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.763195 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="b880c081-4957-4151-9e1f-fb8f397dedb3" containerName="istio-proxy" Nov 02 01:50:25.764148 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.763204 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="819ae12d-e493-42d0-a549-811fb5fe7499" containerName="istio-proxy" Nov 02 01:50:25.764148 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.763215 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="f67c954a-ce5b-41d9-8d99-7d126bd89d59" containerName="main" Nov 02 01:50:25.764148 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.763222 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="c37ba466-a830-4246-832b-a8a4d0b5ab9f" containerName="istio-proxy" Nov 02 01:50:25.764148 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.763232 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="fca29702-c34c-4933-a93a-143c5481ba3a" containerName="payload-pre-processing" Nov 02 01:50:25.764148 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.763240 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5cb7660-e2aa-48a2-9eac-508cd83f93cb" containerName="istio-proxy" Nov 02 01:50:25.764148 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.763248 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="317a9ada-6f39-4dee-8945-9cc2dd1c18eb" containerName="payload-pre-processing" Nov 02 01:50:25.764148 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.763256 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="b880c081-4957-4151-9e1f-fb8f397dedb3" containerName="istio-proxy" Nov 02 01:50:25.764148 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.763263 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4ae0b1f-50f6-46ff-8a9c-0dc3216265cb" containerName="main" Nov 02 01:50:25.764148 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.763270 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d6df883-72f0-4727-acbc-72bb9bd02f05" containerName="istio-proxy" Nov 02 01:50:25.764148 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.763278 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f9b0669-291d-43fa-9511-d74948b03684" containerName="istio-proxy" Nov 02 01:50:25.764148 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.763287 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcf15fa4-15f1-4dc9-8755-9bcccfdc846b" containerName="manager" Nov 02 01:50:25.764148 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.763295 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="392681ce-fe14-4518-9697-efb94f82efdc" containerName="istio-proxy" Nov 02 01:50:25.764148 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.763303 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="d133c468-9c0c-4d15-8b0e-9292b27944f6" containerName="payload-processing" Nov 02 01:50:25.764148 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.763311 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f9b0669-291d-43fa-9511-d74948b03684" containerName="istio-proxy" Nov 02 01:50:25.764148 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.763321 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb3258a9-adfb-46de-a0ab-7b4649948689" containerName="istio-proxy" Nov 02 01:50:25.764148 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.763328 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="17e05811-e6f0-4c5b-be43-d39429d1af11" containerName="payload-pre-processing" Nov 02 01:50:25.764148 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.763336 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="99cc3c61-f501-4a64-8f7e-0f41058aac68" containerName="istio-proxy" Nov 02 01:50:25.764148 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.763343 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="aceec71b-ffcb-47e7-964e-84bc8405f2de" containerName="payload-pre-processing" Nov 02 01:50:25.764148 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.763353 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c056798-2cf8-47b1-9829-67cebda7a248" containerName="main" Nov 02 01:50:25.764148 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.763360 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="79fb94dd-4426-43ed-af58-cfa762e77b13" containerName="payload-processing" Nov 02 01:50:25.764148 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.763392 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="f12da917-8a8f-445b-b939-0bab19110258" containerName="maas-api" Nov 02 01:50:25.764148 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.763401 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="7085c0ab-7ee3-4665-abad-2977c25f68ba" containerName="main" Nov 02 01:50:25.764148 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.763409 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="af28d18e-199e-4c22-bb6d-9b83e7e0473a" containerName="revoke-keys" Nov 02 01:50:25.764148 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.763418 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="f31dc5a1-ea54-4858-ac4c-2b990192ee4c" containerName="istio-proxy" Nov 02 01:50:25.766925 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.766904 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kgg42" Nov 02 01:50:25.781684 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.781595 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kgg42"] Nov 02 01:50:25.904947 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.904907 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzh4j\" (UniqueName: \"kubernetes.io/projected/7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a-kube-api-access-qzh4j\") pod \"maas-default-gateway-openshift-default-76597d6855-kgg42\" (UID: \"7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kgg42" Nov 02 01:50:25.904947 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.904952 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-kgg42\" (UID: \"7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kgg42" Nov 02 01:50:25.905206 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.905026 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-kgg42\" (UID: \"7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kgg42" Nov 02 01:50:25.905206 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.905046 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-kgg42\" (UID: \"7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kgg42" Nov 02 01:50:25.905206 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.905073 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-kgg42\" (UID: \"7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kgg42" Nov 02 01:50:25.905206 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.905091 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-kgg42\" (UID: \"7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kgg42" Nov 02 01:50:25.905431 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.905246 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-kgg42\" (UID: \"7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kgg42" Nov 02 01:50:25.905431 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.905293 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-kgg42\" (UID: \"7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kgg42" Nov 02 01:50:25.905431 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:25.905345 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-kgg42\" (UID: \"7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kgg42" Nov 02 01:50:26.006767 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:26.006731 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzh4j\" (UniqueName: \"kubernetes.io/projected/7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a-kube-api-access-qzh4j\") pod \"maas-default-gateway-openshift-default-76597d6855-kgg42\" (UID: \"7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kgg42" Nov 02 01:50:26.006767 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:26.006773 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-kgg42\" (UID: \"7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kgg42" Nov 02 01:50:26.007002 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:26.006893 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-kgg42\" (UID: \"7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kgg42" Nov 02 01:50:26.007002 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:26.006942 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-kgg42\" (UID: \"7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kgg42" Nov 02 01:50:26.007002 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:26.006996 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-kgg42\" (UID: \"7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kgg42" Nov 02 01:50:26.007143 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:26.007021 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-kgg42\" (UID: \"7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kgg42" Nov 02 01:50:26.007143 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:26.007053 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-kgg42\" (UID: \"7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kgg42" Nov 02 01:50:26.007143 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:26.007077 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-kgg42\" (UID: \"7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kgg42" Nov 02 01:50:26.007303 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:26.007211 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-kgg42\" (UID: \"7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kgg42" Nov 02 01:50:26.007303 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:26.007247 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-kgg42\" (UID: \"7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kgg42" Nov 02 01:50:26.007465 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:26.007135 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-kgg42\" (UID: \"7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kgg42" Nov 02 01:50:26.007533 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:26.007459 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-kgg42\" (UID: \"7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kgg42" Nov 02 01:50:26.007583 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:26.007534 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-kgg42\" (UID: \"7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kgg42" Nov 02 01:50:26.007744 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:26.007722 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-kgg42\" (UID: \"7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kgg42" Nov 02 01:50:26.009426 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:26.009404 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-kgg42\" (UID: \"7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kgg42" Nov 02 01:50:26.009644 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:26.009617 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-kgg42\" (UID: \"7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kgg42" Nov 02 01:50:26.019668 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:26.019616 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-kgg42\" (UID: \"7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kgg42" Nov 02 01:50:26.019756 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:26.019699 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzh4j\" (UniqueName: \"kubernetes.io/projected/7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a-kube-api-access-qzh4j\") pod \"maas-default-gateway-openshift-default-76597d6855-kgg42\" (UID: \"7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kgg42" Nov 02 01:50:26.083604 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:26.083561 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kgg42" Nov 02 01:50:26.366033 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:26.366006 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kgg42"] Nov 02 01:50:26.367821 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:50:26.367797 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b60c37f_0bbb_4e04_8ee7_1c3cad77dc0a.slice/crio-d44f2a42e7de8c4f62f66fa13c0e459105010b72f05c3b541f888f9e54c5a8eb WatchSource:0}: Error finding container d44f2a42e7de8c4f62f66fa13c0e459105010b72f05c3b541f888f9e54c5a8eb: Status 404 returned error can't find the container with id d44f2a42e7de8c4f62f66fa13c0e459105010b72f05c3b541f888f9e54c5a8eb Nov 02 01:50:26.370001 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:26.369969 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:50:26.370116 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:26.370051 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:50:26.370116 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:26.370094 4427 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 01:50:26.475547 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:26.475508 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kgg42" event={"ID":"7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a","Type":"ContainerStarted","Data":"51f99cd95af24ffa0294320a1e88ac91fae07d363607b99ccb7d685f442fcc39"} Nov 02 01:50:26.475744 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:26.475593 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kgg42" event={"ID":"7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a","Type":"ContainerStarted","Data":"d44f2a42e7de8c4f62f66fa13c0e459105010b72f05c3b541f888f9e54c5a8eb"} Nov 02 01:50:26.500429 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:26.500328 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kgg42" podStartSLOduration=1.500306767 podStartE2EDuration="1.500306767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:50:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:50:26.498456859 +0000 UTC m=+2131.119462073" watchObservedRunningTime="2026-11-02 01:50:26.500306767 +0000 UTC m=+2131.121312061" Nov 02 01:50:27.084629 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:27.084588 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kgg42" Nov 02 01:50:27.086269 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:27.086236 4427 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-kgg42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Nov 02 01:50:27.086425 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:27.086295 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kgg42" podUID="7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Nov 02 01:50:28.084796 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:28.084761 4427 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-kgg42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Nov 02 01:50:28.085191 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:28.084815 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kgg42" podUID="7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Nov 02 01:50:29.084842 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:29.084804 4427 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-kgg42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Nov 02 01:50:29.085236 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:29.084863 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kgg42" podUID="7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Nov 02 01:50:30.084840 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:30.084798 4427 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-kgg42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Nov 02 01:50:30.085044 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:30.084852 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kgg42" podUID="7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Nov 02 01:50:31.084907 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:31.084864 4427 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-kgg42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Nov 02 01:50:31.085108 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:31.084934 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kgg42" podUID="7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Nov 02 01:50:31.609162 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:50:31.609118 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:50:32.084916 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:32.084814 4427 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-kgg42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Nov 02 01:50:32.084916 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:32.084883 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kgg42" podUID="7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Nov 02 01:50:33.085005 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:33.084962 4427 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-kgg42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Nov 02 01:50:33.085454 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:33.085036 4427 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kgg42" podUID="7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Nov 02 01:50:34.582239 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:34.582200 4427 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kgg42" Nov 02 01:50:34.583363 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:34.583338 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kgg42" Nov 02 01:50:34.584185 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:34.584164 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kgg42" Nov 02 01:50:49.479630 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:49.479597 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hwzvb"] Nov 02 01:50:49.480191 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:49.479916 4427 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hwzvb" podUID="547c9df8-080d-499b-87d1-69d009750d49" containerName="discovery" containerID="cri-o://241596ab88afd0104e14692beb122a2a882c9e7a2b15c76ebb6cdd8f9f2401e6" gracePeriod=30 Nov 02 01:50:49.601142 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:49.601109 4427 generic.go:356] "Generic (PLEG): container finished" podID="547c9df8-080d-499b-87d1-69d009750d49" containerID="241596ab88afd0104e14692beb122a2a882c9e7a2b15c76ebb6cdd8f9f2401e6" exitCode=0 Nov 02 01:50:49.601299 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:49.601185 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hwzvb" event={"ID":"547c9df8-080d-499b-87d1-69d009750d49","Type":"ContainerDied","Data":"241596ab88afd0104e14692beb122a2a882c9e7a2b15c76ebb6cdd8f9f2401e6"} Nov 02 01:50:49.730238 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:49.730181 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hwzvb" Nov 02 01:50:49.824302 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:49.824265 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/547c9df8-080d-499b-87d1-69d009750d49-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/547c9df8-080d-499b-87d1-69d009750d49-istio-kubeconfig\") pod \"547c9df8-080d-499b-87d1-69d009750d49\" (UID: \"547c9df8-080d-499b-87d1-69d009750d49\") " Nov 02 01:50:49.824529 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:49.824348 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/547c9df8-080d-499b-87d1-69d009750d49-cacerts\" (UniqueName: \"kubernetes.io/secret/547c9df8-080d-499b-87d1-69d009750d49-cacerts\") pod \"547c9df8-080d-499b-87d1-69d009750d49\" (UID: \"547c9df8-080d-499b-87d1-69d009750d49\") " Nov 02 01:50:49.824529 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:49.824397 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/547c9df8-080d-499b-87d1-69d009750d49-istio-token\" (UniqueName: \"kubernetes.io/projected/547c9df8-080d-499b-87d1-69d009750d49-istio-token\") pod \"547c9df8-080d-499b-87d1-69d009750d49\" (UID: \"547c9df8-080d-499b-87d1-69d009750d49\") " Nov 02 01:50:49.824529 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:49.824432 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/547c9df8-080d-499b-87d1-69d009750d49-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/547c9df8-080d-499b-87d1-69d009750d49-istio-csr-ca-configmap\") pod \"547c9df8-080d-499b-87d1-69d009750d49\" (UID: \"547c9df8-080d-499b-87d1-69d009750d49\") " Nov 02 01:50:49.824529 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:49.824451 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/547c9df8-080d-499b-87d1-69d009750d49-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/547c9df8-080d-499b-87d1-69d009750d49-local-certs\") pod \"547c9df8-080d-499b-87d1-69d009750d49\" (UID: \"547c9df8-080d-499b-87d1-69d009750d49\") " Nov 02 01:50:49.824529 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:49.824496 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/547c9df8-080d-499b-87d1-69d009750d49-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/547c9df8-080d-499b-87d1-69d009750d49-istio-csr-dns-cert\") pod \"547c9df8-080d-499b-87d1-69d009750d49\" (UID: \"547c9df8-080d-499b-87d1-69d009750d49\") " Nov 02 01:50:49.824529 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:49.824529 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/547c9df8-080d-499b-87d1-69d009750d49-kube-api-access-lh9qz\" (UniqueName: \"kubernetes.io/projected/547c9df8-080d-499b-87d1-69d009750d49-kube-api-access-lh9qz\") pod \"547c9df8-080d-499b-87d1-69d009750d49\" (UID: \"547c9df8-080d-499b-87d1-69d009750d49\") " Nov 02 01:50:49.824879 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:49.824851 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/547c9df8-080d-499b-87d1-69d009750d49-istio-csr-ca-configmap" pod "547c9df8-080d-499b-87d1-69d009750d49" (UID: "547c9df8-080d-499b-87d1-69d009750d49"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:50:49.826925 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:49.826888 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/547c9df8-080d-499b-87d1-69d009750d49-istio-kubeconfig" pod "547c9df8-080d-499b-87d1-69d009750d49" (UID: "547c9df8-080d-499b-87d1-69d009750d49"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:50:49.826925 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:49.826904 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/547c9df8-080d-499b-87d1-69d009750d49-local-certs" pod "547c9df8-080d-499b-87d1-69d009750d49" (UID: "547c9df8-080d-499b-87d1-69d009750d49"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:50:49.827261 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:49.827240 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/547c9df8-080d-499b-87d1-69d009750d49-istio-token" pod "547c9df8-080d-499b-87d1-69d009750d49" (UID: "547c9df8-080d-499b-87d1-69d009750d49"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:50:49.827334 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:49.827278 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/547c9df8-080d-499b-87d1-69d009750d49-cacerts" pod "547c9df8-080d-499b-87d1-69d009750d49" (UID: "547c9df8-080d-499b-87d1-69d009750d49"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:50:49.827441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:49.827349 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/547c9df8-080d-499b-87d1-69d009750d49-istio-csr-dns-cert" pod "547c9df8-080d-499b-87d1-69d009750d49" (UID: "547c9df8-080d-499b-87d1-69d009750d49"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:50:49.827441 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:49.827398 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/547c9df8-080d-499b-87d1-69d009750d49-kube-api-access-lh9qz" pod "547c9df8-080d-499b-87d1-69d009750d49" (UID: "547c9df8-080d-499b-87d1-69d009750d49"). InnerVolumeSpecName "kube-api-access-lh9qz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:50:49.926168 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:49.926130 4427 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/547c9df8-080d-499b-87d1-69d009750d49-istio-token\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:50:49.926168 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:49.926161 4427 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/547c9df8-080d-499b-87d1-69d009750d49-istio-csr-ca-configmap\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:50:49.926168 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:49.926174 4427 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/547c9df8-080d-499b-87d1-69d009750d49-local-certs\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:50:49.926455 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:49.926182 4427 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/547c9df8-080d-499b-87d1-69d009750d49-istio-csr-dns-cert\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:50:49.926455 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:49.926191 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh9qz\" (UniqueName: \"kubernetes.io/projected/547c9df8-080d-499b-87d1-69d009750d49-kube-api-access-lh9qz\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:50:49.926455 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:49.926199 4427 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/547c9df8-080d-499b-87d1-69d009750d49-istio-kubeconfig\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:50:49.926455 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:49.926207 4427 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/547c9df8-080d-499b-87d1-69d009750d49-cacerts\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:50:50.607942 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:50.607918 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hwzvb" Nov 02 01:50:50.608414 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:50.607917 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hwzvb" event={"ID":"547c9df8-080d-499b-87d1-69d009750d49","Type":"ContainerDied","Data":"b7c4e0320896ee27c0fcf80f5bf9f9b391ed1de194597252aa0fdd18cb62a5b6"} Nov 02 01:50:50.608414 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:50.608040 4427 scope.go:122] "RemoveContainer" containerID="241596ab88afd0104e14692beb122a2a882c9e7a2b15c76ebb6cdd8f9f2401e6" Nov 02 01:50:50.630427 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:50.630402 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hwzvb"] Nov 02 01:50:50.636467 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:50.636446 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hwzvb"] Nov 02 01:50:51.532407 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:51.532361 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="547c9df8-080d-499b-87d1-69d009750d49" path="/var/lib/kubelet/pods/547c9df8-080d-499b-87d1-69d009750d49/volumes" Nov 02 01:50:58.332688 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:50:58.332656 4427 scope.go:122] "RemoveContainer" containerID="dc23a0c381202a1df8b4d96ca6779deadd44781a45c5207c9f4eb9e8333d1e55" Nov 02 01:51:01.581362 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:51:01.581314 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:51:31.580669 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:51:31.580631 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:51:33.024749 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:51:33.024712 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-768858fd-hfnqf"] Nov 02 01:51:33.028062 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:51:33.028021 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-768858fd-hfnqf"] Nov 02 01:51:33.531622 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:51:33.531579 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c58e25b-d880-4738-98fc-2c7948a6153f" path="/var/lib/kubelet/pods/1c58e25b-d880-4738-98fc-2c7948a6153f/volumes" Nov 02 01:51:35.022601 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:51:35.022565 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-068f2e65-dc9v4"] Nov 02 01:51:35.027895 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:51:35.027869 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-068f2e65-dc9v4"] Nov 02 01:51:35.532031 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:51:35.531998 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93282bc6-f0c6-48ce-bfac-998647223d42" path="/var/lib/kubelet/pods/93282bc6-f0c6-48ce-bfac-998647223d42/volumes" Nov 02 01:52:01.581278 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:52:01.581238 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:52:31.585073 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:52:31.585021 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:52:33.087533 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:52:33.087490 4427 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-7l9hx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 01:52:33.088178 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:52:33.088142 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7l9hx" podUID="2a6ab889-0f65-45fa-b105-5d5240243eb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:52:51.024209 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:52:51.024168 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0584dd92-5vd79"] Nov 02 01:52:51.026773 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:52:51.026735 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0584dd92-5vd79"] Nov 02 01:52:51.532654 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:52:51.532619 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="656ff417-fb98-487e-9929-aa0b5c00d091" path="/var/lib/kubelet/pods/656ff417-fb98-487e-9929-aa0b5c00d091/volumes" Nov 02 01:52:58.371889 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:52:58.371854 4427 scope.go:122] "RemoveContainer" containerID="6fdb2b1f725f2a69660139f20952de6e02c54930e2b68416f017210b49d20684" Nov 02 01:52:58.384129 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:52:58.384100 4427 scope.go:122] "RemoveContainer" containerID="b9a3277d2256fe4b6a16984a340fd19b7a21f7eb6724b4912989a4d1bba47422" Nov 02 01:52:58.392656 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:52:58.392637 4427 scope.go:122] "RemoveContainer" containerID="90c6716482c2870b8f0fbbd748a3faea2cb9fd41f481ab5a32a0afc1fb0aeffd" Nov 02 01:52:58.400529 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:52:58.400505 4427 scope.go:122] "RemoveContainer" containerID="3cb32bdf67dd76179964c43a8a2dc21607db46c936c34641f677737d36a0e7b5" Nov 02 01:52:58.408082 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:52:58.408060 4427 scope.go:122] "RemoveContainer" containerID="b9341ce379d985ebf88e474ca879ace52814c24ea3f1db4e267a7e514325e0d4" Nov 02 01:53:01.581862 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:53:01.581826 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:53:02.022180 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:53:02.022135 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc1a1c55-xjk8t"] Nov 02 01:53:02.024626 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:53:02.024599 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc1a1c55-xjk8t"] Nov 02 01:53:03.531455 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:53:03.531423 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b92c9725-5685-4fe0-94b7-53225178c84f" path="/var/lib/kubelet/pods/b92c9725-5685-4fe0-94b7-53225178c84f/volumes" Nov 02 01:53:11.022697 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:53:11.022587 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae252552-ffm4k"] Nov 02 01:53:11.030553 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:53:11.030516 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae252552-ffm4k"] Nov 02 01:53:11.537232 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:53:11.537192 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8ae4e73-de49-4950-a87b-7c08cd7cfe7f" path="/var/lib/kubelet/pods/c8ae4e73-de49-4950-a87b-7c08cd7cfe7f/volumes" Nov 02 01:53:21.023276 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:53:21.023233 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75ad5e26-rhcrt"] Nov 02 01:53:21.025905 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:53:21.025877 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75ad5e26-rhcrt"] Nov 02 01:53:21.532020 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:53:21.531988 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b8eef49-4f84-4080-9006-27b77583d1ea" path="/var/lib/kubelet/pods/7b8eef49-4f84-4080-9006-27b77583d1ea/volumes" Nov 02 01:53:31.581217 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:53:31.581175 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:53:58.441067 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:53:58.441031 4427 scope.go:122] "RemoveContainer" containerID="59cdae7a95e0dbfec6043083207e44d95517a4f385f688d68f9315068d15dbaa" Nov 02 01:54:01.580170 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:54:01.580126 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:54:31.581726 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:54:31.581678 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:54:58.462787 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:54:58.462752 4427 scope.go:122] "RemoveContainer" containerID="6c07049a2de0f237500a61672ed0d680cfe09741a4ffe3d10d9376903b18c318" Nov 02 01:54:58.474007 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:54:58.473938 4427 scope.go:122] "RemoveContainer" containerID="4f1fa6c5e7219500344e7487e6d1487a3e45ed2f2572005be2769b61f3c18a01" Nov 02 01:54:58.482961 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:54:58.482935 4427 scope.go:122] "RemoveContainer" containerID="4980830abe2e70cc134b74a38c9364b7698dba6eb253f7be4fff836b818aaf67" Nov 02 01:54:58.490655 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:54:58.490629 4427 scope.go:122] "RemoveContainer" containerID="606a3e0ea740a9d8d6fde7f999f1e75627b2a2fb5dc2919ba91e24782f032f6c" Nov 02 01:54:58.501000 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:54:58.500981 4427 scope.go:122] "RemoveContainer" containerID="7735d0fe7e4c1230cae26a2eba168e1fd41adfd7fc9ee73abd16135f612016fc" Nov 02 01:54:58.509021 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:54:58.508999 4427 scope.go:122] "RemoveContainer" containerID="4b1a44c1996812e92d6d5b5fcbfaef6f393cc6fe8aad3b7cad7f4c3f50bea340" Nov 02 01:55:01.581759 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:55:01.581721 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:55:31.581340 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:55:31.581302 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:55:57.877872 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:55:57.877837 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08183e29-rnz7h"] Nov 02 01:55:57.878408 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:55:57.878190 4427 cpu_manager.go:414] "containerMap: removing container" podUID="bcf15fa4-15f1-4dc9-8755-9bcccfdc846b" containerName="manager" Nov 02 01:55:57.878408 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:55:57.878201 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcf15fa4-15f1-4dc9-8755-9bcccfdc846b" containerName="manager" Nov 02 01:55:57.878408 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:55:57.878220 4427 cpu_manager.go:414] "containerMap: removing container" podUID="1d6df883-72f0-4727-acbc-72bb9bd02f05" containerName="istio-proxy" Nov 02 01:55:57.878408 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:55:57.878224 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d6df883-72f0-4727-acbc-72bb9bd02f05" containerName="istio-proxy" Nov 02 01:55:57.878408 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:55:57.878231 4427 cpu_manager.go:414] "containerMap: removing container" podUID="547c9df8-080d-499b-87d1-69d009750d49" containerName="discovery" Nov 02 01:55:57.878408 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:55:57.878236 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="547c9df8-080d-499b-87d1-69d009750d49" containerName="discovery" Nov 02 01:55:57.878408 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:55:57.878310 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d6df883-72f0-4727-acbc-72bb9bd02f05" containerName="istio-proxy" Nov 02 01:55:57.878408 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:55:57.878317 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="547c9df8-080d-499b-87d1-69d009750d49" containerName="discovery" Nov 02 01:55:57.881288 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:55:57.881272 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08183e29-rnz7h" Nov 02 01:55:57.889016 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:55:57.888986 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 01:55:57.889188 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:55:57.889056 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\"" Nov 02 01:55:57.891694 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:55:57.891673 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08183e29-rnz7h"] Nov 02 01:55:57.916993 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:55:57.916961 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn8hc\" (UniqueName: \"kubernetes.io/projected/6b58fb2d-64c1-4106-ac48-131db89d11ef-kube-api-access-hn8hc\") pod \"maas-api-revoke-sub-08183e29-rnz7h\" (UID: \"6b58fb2d-64c1-4106-ac48-131db89d11ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08183e29-rnz7h" Nov 02 01:55:57.917162 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:55:57.917009 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b58fb2d-64c1-4106-ac48-131db89d11ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08183e29-rnz7h\" (UID: \"6b58fb2d-64c1-4106-ac48-131db89d11ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08183e29-rnz7h" Nov 02 01:55:57.917162 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:55:57.917114 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b58fb2d-64c1-4106-ac48-131db89d11ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08183e29-rnz7h\" (UID: \"6b58fb2d-64c1-4106-ac48-131db89d11ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08183e29-rnz7h" Nov 02 01:55:58.018421 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:55:58.018365 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b58fb2d-64c1-4106-ac48-131db89d11ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08183e29-rnz7h\" (UID: \"6b58fb2d-64c1-4106-ac48-131db89d11ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08183e29-rnz7h" Nov 02 01:55:58.018601 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:55:58.018481 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn8hc\" (UniqueName: \"kubernetes.io/projected/6b58fb2d-64c1-4106-ac48-131db89d11ef-kube-api-access-hn8hc\") pod \"maas-api-revoke-sub-08183e29-rnz7h\" (UID: \"6b58fb2d-64c1-4106-ac48-131db89d11ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08183e29-rnz7h" Nov 02 01:55:58.018601 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:55:58.018517 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b58fb2d-64c1-4106-ac48-131db89d11ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08183e29-rnz7h\" (UID: \"6b58fb2d-64c1-4106-ac48-131db89d11ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08183e29-rnz7h" Nov 02 01:55:58.019005 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:55:58.018978 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b58fb2d-64c1-4106-ac48-131db89d11ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08183e29-rnz7h\" (UID: \"6b58fb2d-64c1-4106-ac48-131db89d11ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08183e29-rnz7h" Nov 02 01:55:58.020979 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:55:58.020955 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b58fb2d-64c1-4106-ac48-131db89d11ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08183e29-rnz7h\" (UID: \"6b58fb2d-64c1-4106-ac48-131db89d11ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08183e29-rnz7h" Nov 02 01:55:58.033194 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:55:58.033169 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn8hc\" (UniqueName: \"kubernetes.io/projected/6b58fb2d-64c1-4106-ac48-131db89d11ef-kube-api-access-hn8hc\") pod \"maas-api-revoke-sub-08183e29-rnz7h\" (UID: \"6b58fb2d-64c1-4106-ac48-131db89d11ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08183e29-rnz7h" Nov 02 01:55:58.195789 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:55:58.195703 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08183e29-rnz7h" Nov 02 01:55:58.473023 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:55:58.472999 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08183e29-rnz7h"] Nov 02 01:55:58.475450 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:55:58.475422 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b58fb2d_64c1_4106_ac48_131db89d11ef.slice/crio-594c4ed17641cb5b0ea19a74173aa6c311cb5d03cfd8d5b7544bddce73fff3d5 WatchSource:0}: Error finding container 594c4ed17641cb5b0ea19a74173aa6c311cb5d03cfd8d5b7544bddce73fff3d5: Status 404 returned error can't find the container with id 594c4ed17641cb5b0ea19a74173aa6c311cb5d03cfd8d5b7544bddce73fff3d5 Nov 02 01:55:59.320581 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:55:59.320522 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08183e29-rnz7h" event={"ID":"6b58fb2d-64c1-4106-ac48-131db89d11ef","Type":"ContainerStarted","Data":"d3d2d445a5f0eca740e30160969c4029005c40ee5458da71567afd22fe47c343"} Nov 02 01:55:59.321099 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:55:59.320660 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08183e29-rnz7h" event={"ID":"6b58fb2d-64c1-4106-ac48-131db89d11ef","Type":"ContainerStarted","Data":"594c4ed17641cb5b0ea19a74173aa6c311cb5d03cfd8d5b7544bddce73fff3d5"} Nov 02 01:55:59.340421 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:55:59.340348 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08183e29-rnz7h" podStartSLOduration=2.340336498 podStartE2EDuration="2.340336498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:55:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:55:59.339141181 +0000 UTC m=+2463.960146393" watchObservedRunningTime="2026-11-02 01:55:59.340336498 +0000 UTC m=+2463.961341709" Nov 02 01:56:00.326450 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:56:00.326413 4427 generic.go:356] "Generic (PLEG): container finished" podID="6b58fb2d-64c1-4106-ac48-131db89d11ef" containerID="d3d2d445a5f0eca740e30160969c4029005c40ee5458da71567afd22fe47c343" exitCode=0 Nov 02 01:56:00.326868 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:56:00.326480 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08183e29-rnz7h" event={"ID":"6b58fb2d-64c1-4106-ac48-131db89d11ef","Type":"ContainerDied","Data":"d3d2d445a5f0eca740e30160969c4029005c40ee5458da71567afd22fe47c343"} Nov 02 01:56:01.467132 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:56:01.467109 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08183e29-rnz7h" Nov 02 01:56:01.552161 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:56:01.552131 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b58fb2d-64c1-4106-ac48-131db89d11ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b58fb2d-64c1-4106-ac48-131db89d11ef-maas-api-xks-ca\") pod \"6b58fb2d-64c1-4106-ac48-131db89d11ef\" (UID: \"6b58fb2d-64c1-4106-ac48-131db89d11ef\") " Nov 02 01:56:01.552337 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:56:01.552190 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b58fb2d-64c1-4106-ac48-131db89d11ef-kube-api-access-hn8hc\" (UniqueName: \"kubernetes.io/projected/6b58fb2d-64c1-4106-ac48-131db89d11ef-kube-api-access-hn8hc\") pod \"6b58fb2d-64c1-4106-ac48-131db89d11ef\" (UID: \"6b58fb2d-64c1-4106-ac48-131db89d11ef\") " Nov 02 01:56:01.552337 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:56:01.552239 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b58fb2d-64c1-4106-ac48-131db89d11ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b58fb2d-64c1-4106-ac48-131db89d11ef-maas-api-openshift-ca\") pod \"6b58fb2d-64c1-4106-ac48-131db89d11ef\" (UID: \"6b58fb2d-64c1-4106-ac48-131db89d11ef\") " Nov 02 01:56:01.552691 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:56:01.552649 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b58fb2d-64c1-4106-ac48-131db89d11ef-maas-api-openshift-ca" pod "6b58fb2d-64c1-4106-ac48-131db89d11ef" (UID: "6b58fb2d-64c1-4106-ac48-131db89d11ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:56:01.554562 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:56:01.554533 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b58fb2d-64c1-4106-ac48-131db89d11ef-kube-api-access-hn8hc" pod "6b58fb2d-64c1-4106-ac48-131db89d11ef" (UID: "6b58fb2d-64c1-4106-ac48-131db89d11ef"). InnerVolumeSpecName "kube-api-access-hn8hc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:56:01.554665 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:56:01.554568 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b58fb2d-64c1-4106-ac48-131db89d11ef-maas-api-xks-ca" pod "6b58fb2d-64c1-4106-ac48-131db89d11ef" (UID: "6b58fb2d-64c1-4106-ac48-131db89d11ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:56:01.581131 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:56:01.581058 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:56:01.652887 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:56:01.652851 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn8hc\" (UniqueName: \"kubernetes.io/projected/6b58fb2d-64c1-4106-ac48-131db89d11ef-kube-api-access-hn8hc\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:56:01.652887 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:56:01.652882 4427 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b58fb2d-64c1-4106-ac48-131db89d11ef-maas-api-openshift-ca\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:56:01.652887 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:56:01.652897 4427 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b58fb2d-64c1-4106-ac48-131db89d11ef-maas-api-xks-ca\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:56:02.336478 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:56:02.336449 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08183e29-rnz7h" Nov 02 01:56:02.336478 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:56:02.336453 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08183e29-rnz7h" event={"ID":"6b58fb2d-64c1-4106-ac48-131db89d11ef","Type":"ContainerDied","Data":"594c4ed17641cb5b0ea19a74173aa6c311cb5d03cfd8d5b7544bddce73fff3d5"} Nov 02 01:56:02.336681 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:56:02.336527 4427 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="594c4ed17641cb5b0ea19a74173aa6c311cb5d03cfd8d5b7544bddce73fff3d5" Nov 02 01:56:08.882738 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:56:08.882656 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-639dda-scszk"] Nov 02 01:56:08.883222 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:56:08.883129 4427 cpu_manager.go:414] "containerMap: removing container" podUID="6b58fb2d-64c1-4106-ac48-131db89d11ef" containerName="revoke-keys" Nov 02 01:56:08.883222 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:56:08.883144 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b58fb2d-64c1-4106-ac48-131db89d11ef" containerName="revoke-keys" Nov 02 01:56:08.883222 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:56:08.883211 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b58fb2d-64c1-4106-ac48-131db89d11ef" containerName="revoke-keys" Nov 02 01:56:08.886561 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:56:08.886539 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-639dda-scszk" Nov 02 01:56:08.891329 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:56:08.891307 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\"" Nov 02 01:56:08.892862 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:56:08.892816 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 01:56:08.895444 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:56:08.895418 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-639dda-scszk"] Nov 02 01:56:08.924028 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:56:08.923995 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2309ac68-1ae2-4273-8f63-9b68f9dc7648-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-639dda-scszk\" (UID: \"2309ac68-1ae2-4273-8f63-9b68f9dc7648\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-639dda-scszk" Nov 02 01:56:08.924200 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:56:08.924035 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd7rf\" (UniqueName: \"kubernetes.io/projected/2309ac68-1ae2-4273-8f63-9b68f9dc7648-kube-api-access-zd7rf\") pod \"maas-api-revoke-keys-e2e-worker-single-main-639dda-scszk\" (UID: \"2309ac68-1ae2-4273-8f63-9b68f9dc7648\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-639dda-scszk" Nov 02 01:56:08.924273 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:56:08.924195 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2309ac68-1ae2-4273-8f63-9b68f9dc7648-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-639dda-scszk\" (UID: \"2309ac68-1ae2-4273-8f63-9b68f9dc7648\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-639dda-scszk" Nov 02 01:56:09.025516 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:56:09.025479 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2309ac68-1ae2-4273-8f63-9b68f9dc7648-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-639dda-scszk\" (UID: \"2309ac68-1ae2-4273-8f63-9b68f9dc7648\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-639dda-scszk" Nov 02 01:56:09.025713 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:56:09.025552 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2309ac68-1ae2-4273-8f63-9b68f9dc7648-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-639dda-scszk\" (UID: \"2309ac68-1ae2-4273-8f63-9b68f9dc7648\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-639dda-scszk" Nov 02 01:56:09.025713 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:56:09.025578 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd7rf\" (UniqueName: \"kubernetes.io/projected/2309ac68-1ae2-4273-8f63-9b68f9dc7648-kube-api-access-zd7rf\") pod \"maas-api-revoke-keys-e2e-worker-single-main-639dda-scszk\" (UID: \"2309ac68-1ae2-4273-8f63-9b68f9dc7648\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-639dda-scszk" Nov 02 01:56:09.026257 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:56:09.026233 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2309ac68-1ae2-4273-8f63-9b68f9dc7648-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-639dda-scszk\" (UID: \"2309ac68-1ae2-4273-8f63-9b68f9dc7648\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-639dda-scszk" Nov 02 01:56:09.028190 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:56:09.028161 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2309ac68-1ae2-4273-8f63-9b68f9dc7648-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-639dda-scszk\" (UID: \"2309ac68-1ae2-4273-8f63-9b68f9dc7648\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-639dda-scszk" Nov 02 01:56:09.063403 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:56:09.063360 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd7rf\" (UniqueName: \"kubernetes.io/projected/2309ac68-1ae2-4273-8f63-9b68f9dc7648-kube-api-access-zd7rf\") pod \"maas-api-revoke-keys-e2e-worker-single-main-639dda-scszk\" (UID: \"2309ac68-1ae2-4273-8f63-9b68f9dc7648\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-639dda-scszk" Nov 02 01:56:09.200405 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:56:09.200303 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-639dda-scszk" Nov 02 01:56:09.473391 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:56:09.473342 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-639dda-scszk"] Nov 02 01:56:09.475612 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:56:09.475582 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2309ac68_1ae2_4273_8f63_9b68f9dc7648.slice/crio-d70d8dd0367ec106ce7b605b79b2a4766b2d814f356a079ea06bcbfe78ebfb82 WatchSource:0}: Error finding container d70d8dd0367ec106ce7b605b79b2a4766b2d814f356a079ea06bcbfe78ebfb82: Status 404 returned error can't find the container with id d70d8dd0367ec106ce7b605b79b2a4766b2d814f356a079ea06bcbfe78ebfb82 Nov 02 01:56:10.375434 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:56:10.375399 4427 generic.go:356] "Generic (PLEG): container finished" podID="2309ac68-1ae2-4273-8f63-9b68f9dc7648" containerID="f592cb620c7524b4dc2a8ed17030d0aa0dbf189b30e8604ed64943ac5f7b58ab" exitCode=0 Nov 02 01:56:10.375775 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:56:10.375443 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-639dda-scszk" event={"ID":"2309ac68-1ae2-4273-8f63-9b68f9dc7648","Type":"ContainerDied","Data":"f592cb620c7524b4dc2a8ed17030d0aa0dbf189b30e8604ed64943ac5f7b58ab"} Nov 02 01:56:10.375775 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:56:10.375491 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-639dda-scszk" event={"ID":"2309ac68-1ae2-4273-8f63-9b68f9dc7648","Type":"ContainerStarted","Data":"d70d8dd0367ec106ce7b605b79b2a4766b2d814f356a079ea06bcbfe78ebfb82"} Nov 02 01:56:11.514971 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:56:11.514934 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-639dda-scszk" Nov 02 01:56:11.550004 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:56:11.549913 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2309ac68-1ae2-4273-8f63-9b68f9dc7648-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2309ac68-1ae2-4273-8f63-9b68f9dc7648-maas-api-xks-ca\") pod \"2309ac68-1ae2-4273-8f63-9b68f9dc7648\" (UID: \"2309ac68-1ae2-4273-8f63-9b68f9dc7648\") " Nov 02 01:56:11.550172 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:56:11.550118 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2309ac68-1ae2-4273-8f63-9b68f9dc7648-kube-api-access-zd7rf\" (UniqueName: \"kubernetes.io/projected/2309ac68-1ae2-4273-8f63-9b68f9dc7648-kube-api-access-zd7rf\") pod \"2309ac68-1ae2-4273-8f63-9b68f9dc7648\" (UID: \"2309ac68-1ae2-4273-8f63-9b68f9dc7648\") " Nov 02 01:56:11.550232 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:56:11.550202 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2309ac68-1ae2-4273-8f63-9b68f9dc7648-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2309ac68-1ae2-4273-8f63-9b68f9dc7648-maas-api-openshift-ca\") pod \"2309ac68-1ae2-4273-8f63-9b68f9dc7648\" (UID: \"2309ac68-1ae2-4273-8f63-9b68f9dc7648\") " Nov 02 01:56:11.550621 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:56:11.550562 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2309ac68-1ae2-4273-8f63-9b68f9dc7648-maas-api-openshift-ca" pod "2309ac68-1ae2-4273-8f63-9b68f9dc7648" (UID: "2309ac68-1ae2-4273-8f63-9b68f9dc7648"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:56:11.552280 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:56:11.552248 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2309ac68-1ae2-4273-8f63-9b68f9dc7648-maas-api-xks-ca" pod "2309ac68-1ae2-4273-8f63-9b68f9dc7648" (UID: "2309ac68-1ae2-4273-8f63-9b68f9dc7648"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:56:11.552465 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:56:11.552439 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2309ac68-1ae2-4273-8f63-9b68f9dc7648-kube-api-access-zd7rf" pod "2309ac68-1ae2-4273-8f63-9b68f9dc7648" (UID: "2309ac68-1ae2-4273-8f63-9b68f9dc7648"). InnerVolumeSpecName "kube-api-access-zd7rf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:56:11.651980 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:56:11.651893 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zd7rf\" (UniqueName: \"kubernetes.io/projected/2309ac68-1ae2-4273-8f63-9b68f9dc7648-kube-api-access-zd7rf\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:56:11.651980 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:56:11.651923 4427 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2309ac68-1ae2-4273-8f63-9b68f9dc7648-maas-api-openshift-ca\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:56:11.651980 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:56:11.651936 4427 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2309ac68-1ae2-4273-8f63-9b68f9dc7648-maas-api-xks-ca\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:56:12.386098 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:56:12.386065 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-639dda-scszk" Nov 02 01:56:12.386266 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:56:12.386091 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-639dda-scszk" event={"ID":"2309ac68-1ae2-4273-8f63-9b68f9dc7648","Type":"ContainerDied","Data":"d70d8dd0367ec106ce7b605b79b2a4766b2d814f356a079ea06bcbfe78ebfb82"} Nov 02 01:56:12.386266 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:56:12.386145 4427 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d70d8dd0367ec106ce7b605b79b2a4766b2d814f356a079ea06bcbfe78ebfb82" Nov 02 01:56:19.500679 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:56:19.500635 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-639dda-scszk"] Nov 02 01:56:19.503481 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:56:19.503459 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-639dda-scszk"] Nov 02 01:56:19.531944 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:56:19.531909 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2309ac68-1ae2-4273-8f63-9b68f9dc7648" path="/var/lib/kubelet/pods/2309ac68-1ae2-4273-8f63-9b68f9dc7648/volumes" Nov 02 01:56:31.581347 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:56:31.581310 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:57:01.579967 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:57:01.579930 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:57:31.580099 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:57:31.580050 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:58:01.580786 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:58:01.580743 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:58:31.581002 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:58:31.580961 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:59:01.581850 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:59:01.581808 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:59:25.236711 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:25.236632 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93857c28-xgfj4"] Nov 02 01:59:25.237178 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:25.236982 4427 cpu_manager.go:414] "containerMap: removing container" podUID="2309ac68-1ae2-4273-8f63-9b68f9dc7648" containerName="revoke-keys" Nov 02 01:59:25.237178 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:25.236993 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2309ac68-1ae2-4273-8f63-9b68f9dc7648" containerName="revoke-keys" Nov 02 01:59:25.237178 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:25.237069 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="2309ac68-1ae2-4273-8f63-9b68f9dc7648" containerName="revoke-keys" Nov 02 01:59:25.239515 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:25.239498 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93857c28-xgfj4" Nov 02 01:59:25.243608 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:25.243585 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 01:59:25.244892 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:25.244874 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\"" Nov 02 01:59:25.249188 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:25.249161 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93857c28-xgfj4"] Nov 02 01:59:25.302820 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:25.302787 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c56mf\" (UniqueName: \"kubernetes.io/projected/d642f864-e419-49a9-a5e8-776f55c9eedd-kube-api-access-c56mf\") pod \"maas-api-revoke-sub-93857c28-xgfj4\" (UID: \"d642f864-e419-49a9-a5e8-776f55c9eedd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93857c28-xgfj4" Nov 02 01:59:25.303034 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:25.302828 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d642f864-e419-49a9-a5e8-776f55c9eedd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93857c28-xgfj4\" (UID: \"d642f864-e419-49a9-a5e8-776f55c9eedd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93857c28-xgfj4" Nov 02 01:59:25.303034 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:25.302860 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d642f864-e419-49a9-a5e8-776f55c9eedd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93857c28-xgfj4\" (UID: \"d642f864-e419-49a9-a5e8-776f55c9eedd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93857c28-xgfj4" Nov 02 01:59:25.403803 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:25.403764 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c56mf\" (UniqueName: \"kubernetes.io/projected/d642f864-e419-49a9-a5e8-776f55c9eedd-kube-api-access-c56mf\") pod \"maas-api-revoke-sub-93857c28-xgfj4\" (UID: \"d642f864-e419-49a9-a5e8-776f55c9eedd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93857c28-xgfj4" Nov 02 01:59:25.403803 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:25.403806 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d642f864-e419-49a9-a5e8-776f55c9eedd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93857c28-xgfj4\" (UID: \"d642f864-e419-49a9-a5e8-776f55c9eedd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93857c28-xgfj4" Nov 02 01:59:25.404016 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:25.403829 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d642f864-e419-49a9-a5e8-776f55c9eedd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93857c28-xgfj4\" (UID: \"d642f864-e419-49a9-a5e8-776f55c9eedd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93857c28-xgfj4" Nov 02 01:59:25.404527 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:25.404505 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d642f864-e419-49a9-a5e8-776f55c9eedd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93857c28-xgfj4\" (UID: \"d642f864-e419-49a9-a5e8-776f55c9eedd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93857c28-xgfj4" Nov 02 01:59:25.406339 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:25.406321 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d642f864-e419-49a9-a5e8-776f55c9eedd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93857c28-xgfj4\" (UID: \"d642f864-e419-49a9-a5e8-776f55c9eedd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93857c28-xgfj4" Nov 02 01:59:25.415967 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:25.415944 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c56mf\" (UniqueName: \"kubernetes.io/projected/d642f864-e419-49a9-a5e8-776f55c9eedd-kube-api-access-c56mf\") pod \"maas-api-revoke-sub-93857c28-xgfj4\" (UID: \"d642f864-e419-49a9-a5e8-776f55c9eedd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93857c28-xgfj4" Nov 02 01:59:25.554528 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:25.554452 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93857c28-xgfj4" Nov 02 01:59:25.832491 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:25.832272 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93857c28-xgfj4"] Nov 02 01:59:25.834800 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:59:25.834771 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd642f864_e419_49a9_a5e8_776f55c9eedd.slice/crio-2643820d5f27bbaaab8a98f53d5762ef6077c378d8834062f0626c27d82163f8 WatchSource:0}: Error finding container 2643820d5f27bbaaab8a98f53d5762ef6077c378d8834062f0626c27d82163f8: Status 404 returned error can't find the container with id 2643820d5f27bbaaab8a98f53d5762ef6077c378d8834062f0626c27d82163f8 Nov 02 01:59:26.283568 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:26.283529 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93857c28-xgfj4" event={"ID":"d642f864-e419-49a9-a5e8-776f55c9eedd","Type":"ContainerStarted","Data":"ea0cf9ca5dae9b623ffd0e92783c8dcce8a2a5a2eddaf7a2068aac3a3a6d2bd6"} Nov 02 01:59:26.283988 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:26.283586 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93857c28-xgfj4" event={"ID":"d642f864-e419-49a9-a5e8-776f55c9eedd","Type":"ContainerStarted","Data":"2643820d5f27bbaaab8a98f53d5762ef6077c378d8834062f0626c27d82163f8"} Nov 02 01:59:26.303645 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:26.303596 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93857c28-xgfj4" podStartSLOduration=1.3035836060000001 podStartE2EDuration="1.303583606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:59:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:59:26.302187197 +0000 UTC m=+2670.923192409" watchObservedRunningTime="2026-11-02 01:59:26.303583606 +0000 UTC m=+2670.924588817" Nov 02 01:59:27.289652 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:27.289619 4427 generic.go:356] "Generic (PLEG): container finished" podID="d642f864-e419-49a9-a5e8-776f55c9eedd" containerID="ea0cf9ca5dae9b623ffd0e92783c8dcce8a2a5a2eddaf7a2068aac3a3a6d2bd6" exitCode=0 Nov 02 01:59:27.290052 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:27.289674 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93857c28-xgfj4" event={"ID":"d642f864-e419-49a9-a5e8-776f55c9eedd","Type":"ContainerDied","Data":"ea0cf9ca5dae9b623ffd0e92783c8dcce8a2a5a2eddaf7a2068aac3a3a6d2bd6"} Nov 02 01:59:28.423251 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:28.423225 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93857c28-xgfj4" Nov 02 01:59:28.534053 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:28.534016 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d642f864-e419-49a9-a5e8-776f55c9eedd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d642f864-e419-49a9-a5e8-776f55c9eedd-maas-api-xks-ca\") pod \"d642f864-e419-49a9-a5e8-776f55c9eedd\" (UID: \"d642f864-e419-49a9-a5e8-776f55c9eedd\") " Nov 02 01:59:28.534231 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:28.534069 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d642f864-e419-49a9-a5e8-776f55c9eedd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d642f864-e419-49a9-a5e8-776f55c9eedd-maas-api-openshift-ca\") pod \"d642f864-e419-49a9-a5e8-776f55c9eedd\" (UID: \"d642f864-e419-49a9-a5e8-776f55c9eedd\") " Nov 02 01:59:28.534231 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:28.534104 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d642f864-e419-49a9-a5e8-776f55c9eedd-kube-api-access-c56mf\" (UniqueName: \"kubernetes.io/projected/d642f864-e419-49a9-a5e8-776f55c9eedd-kube-api-access-c56mf\") pod \"d642f864-e419-49a9-a5e8-776f55c9eedd\" (UID: \"d642f864-e419-49a9-a5e8-776f55c9eedd\") " Nov 02 01:59:28.534539 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:28.534518 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d642f864-e419-49a9-a5e8-776f55c9eedd-maas-api-openshift-ca" pod "d642f864-e419-49a9-a5e8-776f55c9eedd" (UID: "d642f864-e419-49a9-a5e8-776f55c9eedd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:59:28.536402 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:28.536363 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d642f864-e419-49a9-a5e8-776f55c9eedd-maas-api-xks-ca" pod "d642f864-e419-49a9-a5e8-776f55c9eedd" (UID: "d642f864-e419-49a9-a5e8-776f55c9eedd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:59:28.536535 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:28.536519 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d642f864-e419-49a9-a5e8-776f55c9eedd-kube-api-access-c56mf" pod "d642f864-e419-49a9-a5e8-776f55c9eedd" (UID: "d642f864-e419-49a9-a5e8-776f55c9eedd"). InnerVolumeSpecName "kube-api-access-c56mf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:59:28.635720 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:28.635683 4427 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d642f864-e419-49a9-a5e8-776f55c9eedd-maas-api-openshift-ca\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:59:28.635720 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:28.635717 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c56mf\" (UniqueName: \"kubernetes.io/projected/d642f864-e419-49a9-a5e8-776f55c9eedd-kube-api-access-c56mf\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:59:28.635720 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:28.635727 4427 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d642f864-e419-49a9-a5e8-776f55c9eedd-maas-api-xks-ca\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:59:29.299295 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:29.299264 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93857c28-xgfj4" Nov 02 01:59:29.299553 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:29.299289 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93857c28-xgfj4" event={"ID":"d642f864-e419-49a9-a5e8-776f55c9eedd","Type":"ContainerDied","Data":"2643820d5f27bbaaab8a98f53d5762ef6077c378d8834062f0626c27d82163f8"} Nov 02 01:59:29.299553 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:29.299350 4427 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2643820d5f27bbaaab8a98f53d5762ef6077c378d8834062f0626c27d82163f8" Nov 02 01:59:31.581253 ip-10-0-136-130 kubenswrapper[4427]: E1102 01:59:31.581217 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 01:59:36.118707 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:36.118673 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-693506-p2v8h"] Nov 02 01:59:36.119226 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:36.119047 4427 cpu_manager.go:414] "containerMap: removing container" podUID="d642f864-e419-49a9-a5e8-776f55c9eedd" containerName="revoke-keys" Nov 02 01:59:36.119226 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:36.119059 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d642f864-e419-49a9-a5e8-776f55c9eedd" containerName="revoke-keys" Nov 02 01:59:36.119226 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:36.119168 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="d642f864-e419-49a9-a5e8-776f55c9eedd" containerName="revoke-keys" Nov 02 01:59:36.122975 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:36.122950 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-693506-p2v8h" Nov 02 01:59:36.131473 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:36.131442 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 01:59:36.132729 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:36.132708 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwf5g\"" Nov 02 01:59:36.136194 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:36.136166 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-693506-p2v8h"] Nov 02 01:59:36.206672 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:36.206629 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7hcf\" (UniqueName: \"kubernetes.io/projected/1c97e06b-4782-47db-b009-5c290b84171d-kube-api-access-d7hcf\") pod \"maas-api-revoke-keys-e2e-worker-single-main-693506-p2v8h\" (UID: \"1c97e06b-4782-47db-b009-5c290b84171d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-693506-p2v8h" Nov 02 01:59:36.206858 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:36.206691 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c97e06b-4782-47db-b009-5c290b84171d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-693506-p2v8h\" (UID: \"1c97e06b-4782-47db-b009-5c290b84171d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-693506-p2v8h" Nov 02 01:59:36.206858 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:36.206815 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c97e06b-4782-47db-b009-5c290b84171d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-693506-p2v8h\" (UID: \"1c97e06b-4782-47db-b009-5c290b84171d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-693506-p2v8h" Nov 02 01:59:36.307499 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:36.307463 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c97e06b-4782-47db-b009-5c290b84171d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-693506-p2v8h\" (UID: \"1c97e06b-4782-47db-b009-5c290b84171d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-693506-p2v8h" Nov 02 01:59:36.307713 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:36.307520 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c97e06b-4782-47db-b009-5c290b84171d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-693506-p2v8h\" (UID: \"1c97e06b-4782-47db-b009-5c290b84171d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-693506-p2v8h" Nov 02 01:59:36.307713 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:36.307581 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7hcf\" (UniqueName: \"kubernetes.io/projected/1c97e06b-4782-47db-b009-5c290b84171d-kube-api-access-d7hcf\") pod \"maas-api-revoke-keys-e2e-worker-single-main-693506-p2v8h\" (UID: \"1c97e06b-4782-47db-b009-5c290b84171d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-693506-p2v8h" Nov 02 01:59:36.308223 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:36.308200 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c97e06b-4782-47db-b009-5c290b84171d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-693506-p2v8h\" (UID: \"1c97e06b-4782-47db-b009-5c290b84171d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-693506-p2v8h" Nov 02 01:59:36.310166 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:36.310144 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c97e06b-4782-47db-b009-5c290b84171d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-693506-p2v8h\" (UID: \"1c97e06b-4782-47db-b009-5c290b84171d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-693506-p2v8h" Nov 02 01:59:36.319943 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:36.319922 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7hcf\" (UniqueName: \"kubernetes.io/projected/1c97e06b-4782-47db-b009-5c290b84171d-kube-api-access-d7hcf\") pod \"maas-api-revoke-keys-e2e-worker-single-main-693506-p2v8h\" (UID: \"1c97e06b-4782-47db-b009-5c290b84171d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-693506-p2v8h" Nov 02 01:59:36.441151 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:36.441056 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-693506-p2v8h" Nov 02 01:59:36.718745 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:36.718656 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-693506-p2v8h"] Nov 02 01:59:36.721961 ip-10-0-136-130 kubenswrapper[4427]: W1102 01:59:36.721918 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c97e06b_4782_47db_b009_5c290b84171d.slice/crio-52a71ca9112c8bff61ae6bdaf1be521851d9ad8415bbf2d824892469e773444d WatchSource:0}: Error finding container 52a71ca9112c8bff61ae6bdaf1be521851d9ad8415bbf2d824892469e773444d: Status 404 returned error can't find the container with id 52a71ca9112c8bff61ae6bdaf1be521851d9ad8415bbf2d824892469e773444d Nov 02 01:59:37.337526 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:37.337436 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-693506-p2v8h" event={"ID":"1c97e06b-4782-47db-b009-5c290b84171d","Type":"ContainerStarted","Data":"48490f170cf77fc20e9ba6fbfc166b337fbd8b5c3877bbc0bb6a511b98d56a64"} Nov 02 01:59:37.337526 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:37.337500 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-693506-p2v8h" event={"ID":"1c97e06b-4782-47db-b009-5c290b84171d","Type":"ContainerStarted","Data":"52a71ca9112c8bff61ae6bdaf1be521851d9ad8415bbf2d824892469e773444d"} Nov 02 01:59:37.356926 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:37.356875 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-693506-p2v8h" podStartSLOduration=1.356861334 podStartE2EDuration="1.356861334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:59:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:59:37.354905278 +0000 UTC m=+2681.975910489" watchObservedRunningTime="2026-11-02 01:59:37.356861334 +0000 UTC m=+2681.977866540" Nov 02 01:59:38.343976 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:38.343937 4427 generic.go:356] "Generic (PLEG): container finished" podID="1c97e06b-4782-47db-b009-5c290b84171d" containerID="48490f170cf77fc20e9ba6fbfc166b337fbd8b5c3877bbc0bb6a511b98d56a64" exitCode=0 Nov 02 01:59:38.344404 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:38.344027 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-693506-p2v8h" event={"ID":"1c97e06b-4782-47db-b009-5c290b84171d","Type":"ContainerDied","Data":"48490f170cf77fc20e9ba6fbfc166b337fbd8b5c3877bbc0bb6a511b98d56a64"} Nov 02 01:59:39.482916 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:39.482887 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-693506-p2v8h" Nov 02 01:59:39.539749 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:39.539721 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c97e06b-4782-47db-b009-5c290b84171d-kube-api-access-d7hcf\" (UniqueName: \"kubernetes.io/projected/1c97e06b-4782-47db-b009-5c290b84171d-kube-api-access-d7hcf\") pod \"1c97e06b-4782-47db-b009-5c290b84171d\" (UID: \"1c97e06b-4782-47db-b009-5c290b84171d\") " Nov 02 01:59:39.539930 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:39.539770 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c97e06b-4782-47db-b009-5c290b84171d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c97e06b-4782-47db-b009-5c290b84171d-maas-api-openshift-ca\") pod \"1c97e06b-4782-47db-b009-5c290b84171d\" (UID: \"1c97e06b-4782-47db-b009-5c290b84171d\") " Nov 02 01:59:39.539930 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:39.539807 4427 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c97e06b-4782-47db-b009-5c290b84171d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c97e06b-4782-47db-b009-5c290b84171d-maas-api-xks-ca\") pod \"1c97e06b-4782-47db-b009-5c290b84171d\" (UID: \"1c97e06b-4782-47db-b009-5c290b84171d\") " Nov 02 01:59:39.540179 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:39.540153 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c97e06b-4782-47db-b009-5c290b84171d-maas-api-openshift-ca" pod "1c97e06b-4782-47db-b009-5c290b84171d" (UID: "1c97e06b-4782-47db-b009-5c290b84171d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:59:39.542062 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:39.542034 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c97e06b-4782-47db-b009-5c290b84171d-kube-api-access-d7hcf" pod "1c97e06b-4782-47db-b009-5c290b84171d" (UID: "1c97e06b-4782-47db-b009-5c290b84171d"). InnerVolumeSpecName "kube-api-access-d7hcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:59:39.542270 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:39.542248 4427 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c97e06b-4782-47db-b009-5c290b84171d-maas-api-xks-ca" pod "1c97e06b-4782-47db-b009-5c290b84171d" (UID: "1c97e06b-4782-47db-b009-5c290b84171d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:59:39.640612 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:39.640572 4427 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7hcf\" (UniqueName: \"kubernetes.io/projected/1c97e06b-4782-47db-b009-5c290b84171d-kube-api-access-d7hcf\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:59:39.640612 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:39.640605 4427 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c97e06b-4782-47db-b009-5c290b84171d-maas-api-openshift-ca\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:59:39.640612 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:39.640615 4427 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c97e06b-4782-47db-b009-5c290b84171d-maas-api-xks-ca\") on node \"ip-10-0-136-130.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 01:59:40.354725 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:40.354695 4427 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-693506-p2v8h" Nov 02 01:59:40.354928 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:40.354725 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-693506-p2v8h" event={"ID":"1c97e06b-4782-47db-b009-5c290b84171d","Type":"ContainerDied","Data":"52a71ca9112c8bff61ae6bdaf1be521851d9ad8415bbf2d824892469e773444d"} Nov 02 01:59:40.354928 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:40.354778 4427 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52a71ca9112c8bff61ae6bdaf1be521851d9ad8415bbf2d824892469e773444d" Nov 02 01:59:47.153674 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:47.153634 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-693506-p2v8h"] Nov 02 01:59:47.156506 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:47.156475 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-693506-p2v8h"] Nov 02 01:59:47.533946 ip-10-0-136-130 kubenswrapper[4427]: I1102 01:59:47.533862 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c97e06b-4782-47db-b009-5c290b84171d" path="/var/lib/kubelet/pods/1c97e06b-4782-47db-b009-5c290b84171d/volumes" Nov 02 02:00:01.588983 ip-10-0-136-130 kubenswrapper[4427]: E1102 02:00:01.588934 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 02:00:03.093400 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:00:03.092499 4427 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-7l9hx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 02:00:03.093400 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:00:03.092575 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7l9hx" podUID="2a6ab889-0f65-45fa-b105-5d5240243eb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 02:00:31.580922 ip-10-0-136-130 kubenswrapper[4427]: E1102 02:00:31.580880 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 02:00:35.307889 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:00:35.307834 4427 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-c8lsb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 02:00:35.308712 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:00:35.308676 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-c8lsb" podUID="d77ae499-892d-442f-83de-007ef22dfdcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 02:00:35.644534 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:00:35.644495 4427 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-kgg42 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 02:00:35.644723 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:00:35.644563 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kgg42" podUID="7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 02:00:50.295552 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:00:50.295502 4427 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-c8lsb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 02:00:50.300641 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:00:50.296104 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-c8lsb" podUID="d77ae499-892d-442f-83de-007ef22dfdcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 02:00:50.624640 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:00:50.624586 4427 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-kgg42 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 02:00:50.624865 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:00:50.624664 4427 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-kgg42" podUID="7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 02:01:01.594467 ip-10-0-136-130 kubenswrapper[4427]: E1102 02:01:01.594425 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 02:01:02.029428 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:01:02.029363 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08183e29-rnz7h"] Nov 02 02:01:02.030511 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:01:02.030477 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08183e29-rnz7h"] Nov 02 02:01:03.537757 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:01:03.537721 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b58fb2d-64c1-4106-ac48-131db89d11ef" path="/var/lib/kubelet/pods/6b58fb2d-64c1-4106-ac48-131db89d11ef/volumes" Nov 02 02:01:28.927282 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:01:28.927245 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-694bbcbf67-r9cm6"] Nov 02 02:01:28.927780 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:01:28.927626 4427 cpu_manager.go:414] "containerMap: removing container" podUID="1c97e06b-4782-47db-b009-5c290b84171d" containerName="revoke-keys" Nov 02 02:01:28.927780 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:01:28.927638 4427 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c97e06b-4782-47db-b009-5c290b84171d" containerName="revoke-keys" Nov 02 02:01:28.927780 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:01:28.927709 4427 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c97e06b-4782-47db-b009-5c290b84171d" containerName="revoke-keys" Nov 02 02:01:28.930814 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:01:28.930793 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-694bbcbf67-r9cm6" Nov 02 02:01:28.936036 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:01:28.936013 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 02:01:28.936036 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:01:28.936027 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8qwrd\"" Nov 02 02:01:28.936216 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:01:28.936067 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 02 02:01:28.946471 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:01:28.946448 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-694bbcbf67-r9cm6"] Nov 02 02:01:29.045888 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:01:29.045855 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b088647e-fbb7-462b-9754-231cc665ed0d-openshift-service-ca\") pod \"maas-api-694bbcbf67-r9cm6\" (UID: \"b088647e-fbb7-462b-9754-231cc665ed0d\") " pod="odh-ai-gateway-infra/maas-api-694bbcbf67-r9cm6" Nov 02 02:01:29.045888 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:01:29.045893 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv9hb\" (UniqueName: \"kubernetes.io/projected/b088647e-fbb7-462b-9754-231cc665ed0d-kube-api-access-dv9hb\") pod \"maas-api-694bbcbf67-r9cm6\" (UID: \"b088647e-fbb7-462b-9754-231cc665ed0d\") " pod="odh-ai-gateway-infra/maas-api-694bbcbf67-r9cm6" Nov 02 02:01:29.046163 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:01:29.045931 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b088647e-fbb7-462b-9754-231cc665ed0d-metrics-certs\") pod \"maas-api-694bbcbf67-r9cm6\" (UID: \"b088647e-fbb7-462b-9754-231cc665ed0d\") " pod="odh-ai-gateway-infra/maas-api-694bbcbf67-r9cm6" Nov 02 02:01:29.046163 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:01:29.046063 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b088647e-fbb7-462b-9754-231cc665ed0d-maas-api-tls\") pod \"maas-api-694bbcbf67-r9cm6\" (UID: \"b088647e-fbb7-462b-9754-231cc665ed0d\") " pod="odh-ai-gateway-infra/maas-api-694bbcbf67-r9cm6" Nov 02 02:01:29.147533 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:01:29.147494 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b088647e-fbb7-462b-9754-231cc665ed0d-maas-api-tls\") pod \"maas-api-694bbcbf67-r9cm6\" (UID: \"b088647e-fbb7-462b-9754-231cc665ed0d\") " pod="odh-ai-gateway-infra/maas-api-694bbcbf67-r9cm6" Nov 02 02:01:29.147705 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:01:29.147595 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b088647e-fbb7-462b-9754-231cc665ed0d-openshift-service-ca\") pod \"maas-api-694bbcbf67-r9cm6\" (UID: \"b088647e-fbb7-462b-9754-231cc665ed0d\") " pod="odh-ai-gateway-infra/maas-api-694bbcbf67-r9cm6" Nov 02 02:01:29.147705 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:01:29.147633 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv9hb\" (UniqueName: \"kubernetes.io/projected/b088647e-fbb7-462b-9754-231cc665ed0d-kube-api-access-dv9hb\") pod \"maas-api-694bbcbf67-r9cm6\" (UID: \"b088647e-fbb7-462b-9754-231cc665ed0d\") " pod="odh-ai-gateway-infra/maas-api-694bbcbf67-r9cm6" Nov 02 02:01:29.147705 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:01:29.147693 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b088647e-fbb7-462b-9754-231cc665ed0d-metrics-certs\") pod \"maas-api-694bbcbf67-r9cm6\" (UID: \"b088647e-fbb7-462b-9754-231cc665ed0d\") " pod="odh-ai-gateway-infra/maas-api-694bbcbf67-r9cm6" Nov 02 02:01:29.148351 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:01:29.148321 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b088647e-fbb7-462b-9754-231cc665ed0d-openshift-service-ca\") pod \"maas-api-694bbcbf67-r9cm6\" (UID: \"b088647e-fbb7-462b-9754-231cc665ed0d\") " pod="odh-ai-gateway-infra/maas-api-694bbcbf67-r9cm6" Nov 02 02:01:29.150071 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:01:29.150049 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b088647e-fbb7-462b-9754-231cc665ed0d-maas-api-tls\") pod \"maas-api-694bbcbf67-r9cm6\" (UID: \"b088647e-fbb7-462b-9754-231cc665ed0d\") " pod="odh-ai-gateway-infra/maas-api-694bbcbf67-r9cm6" Nov 02 02:01:29.150191 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:01:29.150174 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b088647e-fbb7-462b-9754-231cc665ed0d-metrics-certs\") pod \"maas-api-694bbcbf67-r9cm6\" (UID: \"b088647e-fbb7-462b-9754-231cc665ed0d\") " pod="odh-ai-gateway-infra/maas-api-694bbcbf67-r9cm6" Nov 02 02:01:29.163121 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:01:29.163090 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv9hb\" (UniqueName: \"kubernetes.io/projected/b088647e-fbb7-462b-9754-231cc665ed0d-kube-api-access-dv9hb\") pod \"maas-api-694bbcbf67-r9cm6\" (UID: \"b088647e-fbb7-462b-9754-231cc665ed0d\") " pod="odh-ai-gateway-infra/maas-api-694bbcbf67-r9cm6" Nov 02 02:01:29.243398 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:01:29.243291 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-694bbcbf67-r9cm6" Nov 02 02:01:29.523877 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:01:29.523849 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-694bbcbf67-r9cm6"] Nov 02 02:01:29.525920 ip-10-0-136-130 kubenswrapper[4427]: W1102 02:01:29.525890 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb088647e_fbb7_462b_9754_231cc665ed0d.slice/crio-7aa4287d5276f50e0769b4adb637ae854871e32aed63f5492438ad83aaff4af3 WatchSource:0}: Error finding container 7aa4287d5276f50e0769b4adb637ae854871e32aed63f5492438ad83aaff4af3: Status 404 returned error can't find the container with id 7aa4287d5276f50e0769b4adb637ae854871e32aed63f5492438ad83aaff4af3 Nov 02 02:01:29.527237 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:01:29.527217 4427 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 02:01:30.002948 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:01:30.002910 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-694bbcbf67-r9cm6" event={"ID":"b088647e-fbb7-462b-9754-231cc665ed0d","Type":"ContainerStarted","Data":"7aa4287d5276f50e0769b4adb637ae854871e32aed63f5492438ad83aaff4af3"} Nov 02 02:01:31.008477 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:01:31.008443 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-694bbcbf67-r9cm6" event={"ID":"b088647e-fbb7-462b-9754-231cc665ed0d","Type":"ContainerStarted","Data":"b1d15465f1a20c9530204fad446c9edb9b8998c9a68d3f01dcdb8143b8228573"} Nov 02 02:01:31.008899 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:01:31.008502 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-694bbcbf67-r9cm6" Nov 02 02:01:31.030034 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:01:31.029985 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-694bbcbf67-r9cm6" podStartSLOduration=2.528452713 podStartE2EDuration="3.029967928s" totalImagesPullingTime="501.515215ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:01:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:01:31.029470413 +0000 UTC m=+2795.650475623" watchObservedRunningTime="2026-11-02 02:01:31.029967928 +0000 UTC m=+2795.650973141" Nov 02 02:01:31.580422 ip-10-0-136-130 kubenswrapper[4427]: E1102 02:01:31.580362 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 02:01:37.020246 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:01:37.020214 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-694bbcbf67-r9cm6" Nov 02 02:01:58.583819 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:01:58.583783 4427 scope.go:122] "RemoveContainer" containerID="d3d2d445a5f0eca740e30160969c4029005c40ee5458da71567afd22fe47c343" Nov 02 02:02:01.581615 ip-10-0-136-130 kubenswrapper[4427]: E1102 02:02:01.581579 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 02:02:31.581789 ip-10-0-136-130 kubenswrapper[4427]: E1102 02:02:31.581702 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 02:02:58.604339 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:02:58.604299 4427 scope.go:122] "RemoveContainer" containerID="f592cb620c7524b4dc2a8ed17030d0aa0dbf189b30e8604ed64943ac5f7b58ab" Nov 02 02:03:01.582703 ip-10-0-136-130 kubenswrapper[4427]: E1102 02:03:01.582667 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 02:03:31.580252 ip-10-0-136-130 kubenswrapper[4427]: E1102 02:03:31.580216 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 02:04:01.582180 ip-10-0-136-130 kubenswrapper[4427]: E1102 02:04:01.582132 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 02:04:29.021823 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:04:29.021785 4427 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93857c28-xgfj4"] Nov 02 02:04:29.024121 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:04:29.024092 4427 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93857c28-xgfj4"] Nov 02 02:04:29.532008 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:04:29.531973 4427 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d642f864-e419-49a9-a5e8-776f55c9eedd" path="/var/lib/kubelet/pods/d642f864-e419-49a9-a5e8-776f55c9eedd/volumes" Nov 02 02:04:31.648464 ip-10-0-136-130 kubenswrapper[4427]: E1102 02:04:31.648422 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 02:05:01.581515 ip-10-0-136-130 kubenswrapper[4427]: E1102 02:05:01.581474 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 02:05:07.036869 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:05:07.036783 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-rrmfw_6e1cb75b-5ed8-4213-a5db-4f60848c2612/manager/0.log" Nov 02 02:05:07.846852 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:05:07.846816 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-45dsr_e8c52127-2f29-46e8-8c99-ce29cfc89614/manager/0.log" Nov 02 02:05:09.422287 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:05:09.422254 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-694bbcbf67-r9cm6_b088647e-fbb7-462b-9754-231cc665ed0d/maas-api/0.log" Nov 02 02:05:11.069347 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:05:11.069314 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-7z6rk_4d514a34-b384-47a4-9193-c475aefc8447/postgres/0.log" Nov 02 02:05:13.263869 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:05:13.263837 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-nr4n2_65197f55-7074-436e-a8b1-320105fde335/kuadrant-console-plugin/0.log" Nov 02 02:05:14.985272 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:05:14.985241 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6c85779f4b-8nj6c_b5f3fd9e-dc3c-4e95-b44d-28565e7a270c/kube-auth-proxy/0.log" Nov 02 02:05:15.750727 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:05:15.750698 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-7l9hx_2a6ab889-0f65-45fa-b105-5d5240243eb8/istio-proxy/0.log" Nov 02 02:05:15.933751 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:05:15.933724 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-c8lsb_d77ae499-892d-442f-83de-007ef22dfdcc/istio-proxy/0.log" Nov 02 02:05:16.284843 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:05:16.284805 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-kgg42_7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a/istio-proxy/0.log" Nov 02 02:05:20.272217 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:05:20.272182 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66_d73ae484-a9b8-4565-a346-0d803a1981ae/main/0.log" Nov 02 02:05:31.180629 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:05:31.180598 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-rrmfw_6e1cb75b-5ed8-4213-a5db-4f60848c2612/manager/0.log" Nov 02 02:05:31.581712 ip-10-0-136-130 kubenswrapper[4427]: E1102 02:05:31.581622 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 02:05:31.820282 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:05:31.820250 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-45dsr_e8c52127-2f29-46e8-8c99-ce29cfc89614/manager/0.log" Nov 02 02:05:32.836805 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:05:32.836775 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-694bbcbf67-r9cm6_b088647e-fbb7-462b-9754-231cc665ed0d/maas-api/0.log" Nov 02 02:05:34.138124 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:05:34.138093 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-7z6rk_4d514a34-b384-47a4-9193-c475aefc8447/postgres/0.log" Nov 02 02:05:35.914603 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:05:35.914573 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-nr4n2_65197f55-7074-436e-a8b1-320105fde335/kuadrant-console-plugin/0.log" Nov 02 02:05:37.435327 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:05:37.435287 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6c85779f4b-8nj6c_b5f3fd9e-dc3c-4e95-b44d-28565e7a270c/kube-auth-proxy/0.log" Nov 02 02:05:38.168514 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:05:38.168487 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-7l9hx_2a6ab889-0f65-45fa-b105-5d5240243eb8/istio-proxy/0.log" Nov 02 02:05:38.347591 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:05:38.347555 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-c8lsb_d77ae499-892d-442f-83de-007ef22dfdcc/istio-proxy/0.log" Nov 02 02:05:38.734445 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:05:38.734416 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-kgg42_7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a/istio-proxy/0.log" Nov 02 02:05:41.979252 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:05:41.979217 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-pfc66_d73ae484-a9b8-4565-a346-0d803a1981ae/main/0.log" Nov 02 02:05:58.638572 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:05:58.638540 4427 scope.go:122] "RemoveContainer" containerID="48490f170cf77fc20e9ba6fbfc166b337fbd8b5c3877bbc0bb6a511b98d56a64" Nov 02 02:05:58.647342 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:05:58.647321 4427 scope.go:122] "RemoveContainer" containerID="ea0cf9ca5dae9b623ffd0e92783c8dcce8a2a5a2eddaf7a2068aac3a3a6d2bd6" Nov 02 02:06:01.581510 ip-10-0-136-130 kubenswrapper[4427]: E1102 02:06:01.581470 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 02:06:31.581629 ip-10-0-136-130 kubenswrapper[4427]: E1102 02:06:31.581592 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 02:07:01.580978 ip-10-0-136-130 kubenswrapper[4427]: E1102 02:07:01.580930 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 02:07:12.819981 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:12.819937 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-kl8v7_9916e33f-7f1d-4cb3-81fc-f4eb8e2bf074/global-pull-secret-syncer/0.log" Nov 02 02:07:12.820540 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:12.820360 4427 ???:1] "http: TLS handshake error from 10.0.136.130:41488: EOF" Nov 02 02:07:13.235186 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:13.235152 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vcsls_16eed615-a448-4416-b06e-1c33d1fffb6d/konnectivity-agent/0.log" Nov 02 02:07:13.264781 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:13.264740 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-130.us-east-2.compute.internal_dabfce80149e41ad659d129c965945bf/haproxy/0.log" Nov 02 02:07:20.240722 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:20.240692 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-nr4n2_65197f55-7074-436e-a8b1-320105fde335/kuadrant-console-plugin/0.log" Nov 02 02:07:22.271030 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:22.270997 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pmztz_685b3025-3cb1-4149-8d5f-4271ecdacf99/node-exporter/0.log" Nov 02 02:07:22.294689 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:22.294663 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pmztz_685b3025-3cb1-4149-8d5f-4271ecdacf99/kube-rbac-proxy/0.log" Nov 02 02:07:22.318519 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:22.318496 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pmztz_685b3025-3cb1-4149-8d5f-4271ecdacf99/init-textfile/0.log" Nov 02 02:07:22.439737 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:22.439704 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-jz8q4_592c7dda-a5ec-4783-9960-3968165d30e6/kube-rbac-proxy-main/0.log" Nov 02 02:07:22.464963 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:22.464936 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-jz8q4_592c7dda-a5ec-4783-9960-3968165d30e6/kube-rbac-proxy-self/0.log" Nov 02 02:07:22.493209 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:22.493185 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-jz8q4_592c7dda-a5ec-4783-9960-3968165d30e6/openshift-state-metrics/0.log" Nov 02 02:07:22.534118 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:22.534037 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2a776f00-fcf7-4609-a76d-5527848f1a13/prometheus/0.log" Nov 02 02:07:22.558064 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:22.558035 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2a776f00-fcf7-4609-a76d-5527848f1a13/config-reloader/0.log" Nov 02 02:07:22.581895 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:22.581864 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2a776f00-fcf7-4609-a76d-5527848f1a13/thanos-sidecar/0.log" Nov 02 02:07:22.606064 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:22.606039 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2a776f00-fcf7-4609-a76d-5527848f1a13/kube-rbac-proxy-web/0.log" Nov 02 02:07:22.634287 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:22.634265 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2a776f00-fcf7-4609-a76d-5527848f1a13/kube-rbac-proxy/0.log" Nov 02 02:07:22.659441 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:22.659415 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2a776f00-fcf7-4609-a76d-5527848f1a13/kube-rbac-proxy-thanos/0.log" Nov 02 02:07:22.685051 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:22.685023 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2a776f00-fcf7-4609-a76d-5527848f1a13/init-config-reloader/0.log" Nov 02 02:07:22.715521 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:22.715480 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-5229n_7965489f-4e02-4318-ad66-64346010ec0c/prometheus-operator/0.log" Nov 02 02:07:22.735874 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:22.735850 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-5229n_7965489f-4e02-4318-ad66-64346010ec0c/kube-rbac-proxy/0.log" Nov 02 02:07:22.885480 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:22.885448 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d6794dd76-m26ph_30927eb5-dc3e-46e8-9211-a9fa05d314cd/thanos-query/0.log" Nov 02 02:07:22.919220 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:22.919195 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d6794dd76-m26ph_30927eb5-dc3e-46e8-9211-a9fa05d314cd/kube-rbac-proxy-web/0.log" Nov 02 02:07:22.944925 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:22.944901 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d6794dd76-m26ph_30927eb5-dc3e-46e8-9211-a9fa05d314cd/kube-rbac-proxy/0.log" Nov 02 02:07:22.972872 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:22.972840 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d6794dd76-m26ph_30927eb5-dc3e-46e8-9211-a9fa05d314cd/prom-label-proxy/0.log" Nov 02 02:07:22.983601 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:22.983576 4427 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zr67k/perf-node-gather-daemonset-qz7mq"] Nov 02 02:07:22.987273 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:22.987254 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-qz7mq" Nov 02 02:07:22.995732 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:22.995712 4427 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zr67k\"/\"default-dockercfg-89bfr\"" Nov 02 02:07:22.995829 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:22.995752 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zr67k\"/\"kube-root-ca.crt\"" Nov 02 02:07:22.995829 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:22.995814 4427 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zr67k\"/\"openshift-service-ca.crt\"" Nov 02 02:07:22.999669 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:22.999647 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zr67k/perf-node-gather-daemonset-qz7mq"] Nov 02 02:07:23.009569 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:23.009551 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d6794dd76-m26ph_30927eb5-dc3e-46e8-9211-a9fa05d314cd/kube-rbac-proxy-rules/0.log" Nov 02 02:07:23.039987 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:23.039960 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d6794dd76-m26ph_30927eb5-dc3e-46e8-9211-a9fa05d314cd/kube-rbac-proxy-metrics/0.log" Nov 02 02:07:23.138024 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:23.137916 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f4e3d103-c5bb-4fe3-8227-939ad64dc7f4-podres\") pod \"perf-node-gather-daemonset-qz7mq\" (UID: \"f4e3d103-c5bb-4fe3-8227-939ad64dc7f4\") " pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-qz7mq" Nov 02 02:07:23.138024 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:23.137982 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f4e3d103-c5bb-4fe3-8227-939ad64dc7f4-proc\") pod \"perf-node-gather-daemonset-qz7mq\" (UID: \"f4e3d103-c5bb-4fe3-8227-939ad64dc7f4\") " pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-qz7mq" Nov 02 02:07:23.138237 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:23.138051 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f4e3d103-c5bb-4fe3-8227-939ad64dc7f4-lib-modules\") pod \"perf-node-gather-daemonset-qz7mq\" (UID: \"f4e3d103-c5bb-4fe3-8227-939ad64dc7f4\") " pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-qz7mq" Nov 02 02:07:23.138237 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:23.138072 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fnj9\" (UniqueName: \"kubernetes.io/projected/f4e3d103-c5bb-4fe3-8227-939ad64dc7f4-kube-api-access-5fnj9\") pod \"perf-node-gather-daemonset-qz7mq\" (UID: \"f4e3d103-c5bb-4fe3-8227-939ad64dc7f4\") " pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-qz7mq" Nov 02 02:07:23.138237 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:23.138180 4427 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f4e3d103-c5bb-4fe3-8227-939ad64dc7f4-sys\") pod \"perf-node-gather-daemonset-qz7mq\" (UID: \"f4e3d103-c5bb-4fe3-8227-939ad64dc7f4\") " pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-qz7mq" Nov 02 02:07:23.238833 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:23.238792 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f4e3d103-c5bb-4fe3-8227-939ad64dc7f4-podres\") pod \"perf-node-gather-daemonset-qz7mq\" (UID: \"f4e3d103-c5bb-4fe3-8227-939ad64dc7f4\") " pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-qz7mq" Nov 02 02:07:23.238833 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:23.238841 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f4e3d103-c5bb-4fe3-8227-939ad64dc7f4-proc\") pod \"perf-node-gather-daemonset-qz7mq\" (UID: \"f4e3d103-c5bb-4fe3-8227-939ad64dc7f4\") " pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-qz7mq" Nov 02 02:07:23.239075 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:23.238870 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f4e3d103-c5bb-4fe3-8227-939ad64dc7f4-lib-modules\") pod \"perf-node-gather-daemonset-qz7mq\" (UID: \"f4e3d103-c5bb-4fe3-8227-939ad64dc7f4\") " pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-qz7mq" Nov 02 02:07:23.239075 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:23.238894 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fnj9\" (UniqueName: \"kubernetes.io/projected/f4e3d103-c5bb-4fe3-8227-939ad64dc7f4-kube-api-access-5fnj9\") pod \"perf-node-gather-daemonset-qz7mq\" (UID: \"f4e3d103-c5bb-4fe3-8227-939ad64dc7f4\") " pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-qz7mq" Nov 02 02:07:23.239075 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:23.238942 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f4e3d103-c5bb-4fe3-8227-939ad64dc7f4-podres\") pod \"perf-node-gather-daemonset-qz7mq\" (UID: \"f4e3d103-c5bb-4fe3-8227-939ad64dc7f4\") " pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-qz7mq" Nov 02 02:07:23.239075 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:23.238945 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f4e3d103-c5bb-4fe3-8227-939ad64dc7f4-proc\") pod \"perf-node-gather-daemonset-qz7mq\" (UID: \"f4e3d103-c5bb-4fe3-8227-939ad64dc7f4\") " pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-qz7mq" Nov 02 02:07:23.239075 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:23.239019 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f4e3d103-c5bb-4fe3-8227-939ad64dc7f4-lib-modules\") pod \"perf-node-gather-daemonset-qz7mq\" (UID: \"f4e3d103-c5bb-4fe3-8227-939ad64dc7f4\") " pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-qz7mq" Nov 02 02:07:23.239075 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:23.239042 4427 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f4e3d103-c5bb-4fe3-8227-939ad64dc7f4-sys\") pod \"perf-node-gather-daemonset-qz7mq\" (UID: \"f4e3d103-c5bb-4fe3-8227-939ad64dc7f4\") " pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-qz7mq" Nov 02 02:07:23.239075 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:23.239069 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f4e3d103-c5bb-4fe3-8227-939ad64dc7f4-sys\") pod \"perf-node-gather-daemonset-qz7mq\" (UID: \"f4e3d103-c5bb-4fe3-8227-939ad64dc7f4\") " pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-qz7mq" Nov 02 02:07:23.251410 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:23.251368 4427 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fnj9\" (UniqueName: \"kubernetes.io/projected/f4e3d103-c5bb-4fe3-8227-939ad64dc7f4-kube-api-access-5fnj9\") pod \"perf-node-gather-daemonset-qz7mq\" (UID: \"f4e3d103-c5bb-4fe3-8227-939ad64dc7f4\") " pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-qz7mq" Nov 02 02:07:23.300497 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:23.300466 4427 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-qz7mq" Nov 02 02:07:23.575668 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:23.575623 4427 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zr67k/perf-node-gather-daemonset-qz7mq"] Nov 02 02:07:23.578097 ip-10-0-136-130 kubenswrapper[4427]: W1102 02:07:23.578063 4427 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf4e3d103_c5bb_4fe3_8227_939ad64dc7f4.slice/crio-98e4ff89bcfa30d1654b0df111f189a33b097c8843d18ccdfd1496946cdcf44c WatchSource:0}: Error finding container 98e4ff89bcfa30d1654b0df111f189a33b097c8843d18ccdfd1496946cdcf44c: Status 404 returned error can't find the container with id 98e4ff89bcfa30d1654b0df111f189a33b097c8843d18ccdfd1496946cdcf44c Nov 02 02:07:23.673791 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:23.673755 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-qz7mq" event={"ID":"f4e3d103-c5bb-4fe3-8227-939ad64dc7f4","Type":"ContainerStarted","Data":"98e4ff89bcfa30d1654b0df111f189a33b097c8843d18ccdfd1496946cdcf44c"} Nov 02 02:07:24.679993 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:24.679953 4427 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-qz7mq" event={"ID":"f4e3d103-c5bb-4fe3-8227-939ad64dc7f4","Type":"ContainerStarted","Data":"d700e281ee0f34b726554b880129cbc1370829c0c809fb2bb2186b120ce8598b"} Nov 02 02:07:24.680469 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:24.680120 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-qz7mq" Nov 02 02:07:24.697794 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:24.697747 4427 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-qz7mq" podStartSLOduration=2.6977316460000003 podStartE2EDuration="2.697731646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 02:07:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:07:24.697161376 +0000 UTC m=+3149.318166588" watchObservedRunningTime="2026-11-02 02:07:24.697731646 +0000 UTC m=+3149.318736858" Nov 02 02:07:27.198063 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:27.198032 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xcm6b_00d17dbb-cb72-4413-9559-4487c313158c/dns/0.log" Nov 02 02:07:27.222169 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:27.222120 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xcm6b_00d17dbb-cb72-4413-9559-4487c313158c/kube-rbac-proxy/0.log" Nov 02 02:07:27.278235 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:27.278185 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qwzn4_77df90af-e163-47ea-83ad-99bc1258f3d5/dns-node-resolver/0.log" Nov 02 02:07:27.982023 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:27.981992 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-596688686c-rwtbd_872bde0f-dfbc-4226-928f-8c6dfd95281a/registry/0.log" Nov 02 02:07:28.006140 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:28.006108 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bgczm_41f79a83-9886-4c59-a904-328463dfe674/node-ca/0.log" Nov 02 02:07:30.696229 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:30.696203 4427 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zr67k/perf-node-gather-daemonset-qz7mq" Nov 02 02:07:31.334533 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:31.334506 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6c85779f4b-8nj6c_b5f3fd9e-dc3c-4e95-b44d-28565e7a270c/kube-auth-proxy/0.log" Nov 02 02:07:31.461822 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:31.461782 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-7l9hx_2a6ab889-0f65-45fa-b105-5d5240243eb8/istio-proxy/0.log" Nov 02 02:07:31.491218 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:31.491192 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-c8lsb_d77ae499-892d-442f-83de-007ef22dfdcc/istio-proxy/0.log" Nov 02 02:07:31.552085 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:31.552056 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-kgg42_7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a/istio-proxy/0.log" Nov 02 02:07:31.580778 ip-10-0-136-130 kubenswrapper[4427]: E1102 02:07:31.580746 4427 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gs62j" containerName="extractor" Nov 02 02:07:34.297763 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:34.297725 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pn4kl_a75a9f45-7678-4014-92d9-93d3f5b260ba/serve-healthcheck-canary/0.log" Nov 02 02:07:35.018680 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:35.018651 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gs62j_23bf37a9-9b32-47f9-a4e0-364a7273c02f/kube-rbac-proxy/0.log" Nov 02 02:07:35.047853 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:35.047804 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gs62j_23bf37a9-9b32-47f9-a4e0-364a7273c02f/exporter/0.log" Nov 02 02:07:35.074833 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:35.074775 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gs62j_23bf37a9-9b32-47f9-a4e0-364a7273c02f/extractor/0.log" Nov 02 02:07:36.031425 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:36.031341 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6c85779f4b-8nj6c_b5f3fd9e-dc3c-4e95-b44d-28565e7a270c/kube-auth-proxy/0.log" Nov 02 02:07:36.104446 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:36.104416 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-7l9hx_2a6ab889-0f65-45fa-b105-5d5240243eb8/istio-proxy/0.log" Nov 02 02:07:36.123190 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:36.123165 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-c8lsb_d77ae499-892d-442f-83de-007ef22dfdcc/istio-proxy/0.log" Nov 02 02:07:36.159790 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:36.159765 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-kgg42_7b60c37f-0bbb-4e04-8ee7-1c3cad77dc0a/istio-proxy/0.log" Nov 02 02:07:37.733485 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:37.733460 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-7d6cbb5df9-zxd5z_2ff1362b-0201-4e45-8aeb-535413957e33/manager/0.log" Nov 02 02:07:37.759834 ip-10-0-136-130 kubenswrapper[4427]: I1102 02:07:37.759805 4427 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-7d6cbb5df9-zxd5z_2ff1362b-0201-4e45-8aeb-535413957e33/copy-manifests/0.log"